Language Modelling with Pixels
Authors: Phillip Rust
Jonas F. Lotz
Emanuele Bugliarello
Elizabeth Salesky
Miryam de Lhoneux
Desmond Elliott
Venue: ICLR 2023 notable top 5% (2023)
Keywords: representation learning
nlp
transformers
language model
masked autoencoder

Abstract: Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented in the embedding matrix and computational issues in the output layer. This paper introduces PIXEL, the Pixel-based Encoder of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches instead of predicting a distribution over tokens. We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts. We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEL is more robust than BERT to orthographic attacks and linguistic code-switching, further confirming the benefits of modelling language with pixels.

Published as a conference paper at ICLR 2023
LANGUAGE MODELLING WITH PIXELS
Phillip Rust' Jonas E. Lotz 1.2
Emanuele Bugliarello !
Elizabeth Salesky² Miryam de Lhoneux Desmond Elliott
University of Copenhagen "ROCKWOOL Foundation Research Unit
JJohns Hopkins University "KU Leuven
Piooner Centre for Al
p. rustrddii... ku. dk
ABSTRACT
Language models are defined over a finite set of inputs, which creates a vocab- ulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck results in a trade-off between what can be represented the embedding matrix and computational issues in the output layer. This paper in- troduces PIXEL, the Fixel-baeed Encoder of Language, which suffers from neither of these issues. PIXEL is pretrained language model that renders text images, making it possible to transfer representations across languages based on ortho- graphic similarity
the co-activation of pixels. PIXEL is trained to reconstruct the pixels of masked patches instead of predicting a distribution over tokens I We pretrain the 86M parameter PIXEL model on the same English data as BERT and evaluate on syntactic and semantic tasks in typologically diverse languages, in- cluding various non-Latin scripts. We find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEI. is slightly weaker than BERT when working with Latin scripts. Furthermore, we find that PIXEI. is more robust than BERT to ortho- graphic attacks and linguistic code-switching, further confirming the benefits modelling language with pixels.
INTRODUCTION
Natural language processing has rapidly progressed in recent years due to a combination of self- supervised representation learning, i.e. pretrained language models (PLMs) like BERT (Devlin et al.) 2019). GPT-3 (Brown et al.] (2020). and XLM-R (Conneau et al., 2020); large unlabelled datasets; such as C4 (Raffel et al., 2020). The Pile (Gao et al.) 2020): and large-scale computing power (Hirschberg & Manning, 2015). Despite this progress, these models only cover fraction of the world's languages, with large ieqqulliiies in performance (Pires et 2019 [ Lauscher et al. 20200) and the majority of languages are falling behind English (Joshi al.) 2020b] Bugliarello et al.) 2022). Even within English, these models struggle when tasked with processing noisy inputs (Sun al. 2020 Eger & Benz 2020 In this paper, we show how to effectively support thousands of written languages in a single model while being robust to variations caused by character-level noise.
Language models typically support finite vocabulary of categorical inputs, e.g. characters, sub- words or even words, and much effort has been devoted to vocabulary construction (Wan) (022). On one end of the spectrum, a vocabulary over words has three problems: (i) it is not possible to encode nut-of-vocabullee
words because they lack an entry in closed vocabulary, c.g. "doxing", (ii) there are too many parameters in the word embedding layer, and relatedly, (iii) the normalising constant for the softmax activation in the output layer is too expensive to compute. On the other end of the spectrum, vocabularies over bytes or characters are much smaller, which leads to increased sequence lengths (Keren et al. 2022). In practice, most current models operate than words but larger than characters: subword units (Sennrich et al. [2016] Kudo 2018). Subwords prevent the problem of extremely large embedding and output layers, and support open vocabulary processing. While this is a practical solution in a monolingual context and for some languages like English, dealing with many languages with a variety of scripts will either result in very large vo- cabulary or trade-off over what is represented within a fixed number of subwords (see $ Taken
inputs smaller
See Appendix A for reconstructions of this abstract.

---

Published as a conference paper at ICLR 2023
True: (1999) / Falsc: 0.001
soflmos
Decoder
MLP
Encoder
Embbeddnng Span patches
Embodking
Projection Posntion Embedding
Projection Embbedding
My c
rejol ferten
pap (us]_4 belendann pullt
blakk
Render Text Image
Render Text
Hy eat c00 enjeye octing sarn ostreal
By cool cas
in bosutical box
Lunch and dinowe.
Full at black beans,
(a) PIXEL pretraining
(b) PIXEL finetuning
Figure 1: Overview PIXEL's architecture. Following He et al. (2022), we use a masked autoen- coder with a ViT architecture and lightweight decoder for pretraining (left). At finetuning time (right), the decoder is replaced by task-specific classification head that sits on top of the encoder.
together, given a language model with finite vocabulary, there is bottleneck two locations: at the level of the encoding of the inputs and at the level of estimating the probability distribution over the vocabulary. We call this the vocabulary bottleneck. language model that can handle thousands of languages needs to deal with this problem.
We propose rethink language modelling as a visual recognition task, removing the need for finite vocabulary. Our proposal is inspired by Saalssky et al. (202T), who showed how to train a machine translation model with "visual text representations" in the encoder instead of subwords. Our Pixel- based Encoder of Language (PIXEL.) is built the Masked Autoencoding Visual Transformer (ViT- MAE; He et al. (022). ViT-MAE is a Transformer based encoder-decoder trained to reconstruct the pixels in masked image patches. PIXEL does not have vocabulary embedding layer; instead, text rendered as a sequence of fixed-sized patches, which are processed using a Vision Transformer encoder (Dosovitskiy al. 2021 PIXEL also does not have an expensive output layer when it reconstructs the pixels of the masked patches. In effect, PIXEL provides a solution to the vocabulary bottleneck without needing the prohibitively long sequences of iharacter-bbsed models.
PIXEL is pretrained on the same data as BERT, given our computational resources. This means that it has encountered only ~0.05% non-English text BBlevins & Zettlemoyer 2022) We evaluate PIXEI. on range of syntactic and semantic tasks in 32 typologically diverse languages across 14 scripts, showing that can rapidly adapt to new languages and unseen scripts. PIXEL also evaluated on its ability to handle noisy text caused by orthographic attacks, where pixel-based encoding is clear improvement over subword- based vocabularies. In lexical code-switching experiments, PIXEL performs on-par with BERT and sometimes outperforms the multilingually pretrained MBERT.
PIXEL is a new type language model that can theoretically support any language that can be typeset by modern computer. We make the implementation, the pretrained model including inter- mediate training checkpoints, and the fine-tuned models freely available for the community
APPROACH
The Pixel-based Encoder of Language, PIXEL., consists of three major components: a text renderer, which draws text as an image: an encoder, which encodes the unmasked regions of the image; and decoder, which reconstructs the masked regions at the pixel level. Figure provides an illustration.
2.1
TEXT RENDERER
The key component of PIXEL a text renderer that takes one or more pieces of text and renders them onto blank RGB image Hx We set height H 16 and width W 8464 and choose
We do not claim that language model designed to support thousands of languages should be pretrained only on English text. We expect that pretraining on an appropriate choice another language or multilinguall may provide more remarkable results. PIXEL represents an initial effort at smaller scale. httpps:: thub. compxplpppppxelllllll
2

---

Published as a conference paper at ICLR 2023
(a) Single sentence: (English with emoji)
My
loves pancakes
and my duck & loves grapes
(b) Sentence pair: (Mandarin, Arabic)
عن bنeا لولا الحلدد
الlaلل في
(c) Word-level:
200+
MXU
UU
AnE MKX hUA
NMA AAMAY AdIAAP> 9A+ f+
(Amharic)
Figure 2: Illustrative examples of our rendered text. PIXEL natively supports most writing systems, colour emoji (a), and complex text layouts such as right-to-left writing and ligatures (b). Black patches serve as separators and end-of-sequence markers. Blank patches to the right of the end- of of-sequence marker are treated as sequence padding. For word-level tasks, horizontal spacing can be added between words (c) so that every patch can be assigned to exactly one word (dotted lines indicate patch boundaries for demonstration).
C= 3 RGB input channels, which is equivalent to square colour image with 368 x 368 resolu- tion and corresponds to a sequence 529 image patches of size 16 x 16 pixels Figure 2 shows examples of text inputs rendered by the text renderer. The renderer supports (a) colour emoji and hieroglyphs seripts, (b) left-to-right and right-to-lefi writing systems, and (c) text that requires liga- tures. Analogous
BERT, a sequence can either contain single paragraph of text use black 16 x 16 patches to serve as separators and end-of-sequence (EOS) markers. Blank (white) patches after the end-of-sequence marker are treated padding by PIXEL, where no attention scores or losses are computed. Sequences longer than the maximum length are either truncated or split into multiple sequences. Further technical details about the renderer are provided in Appendix D
text pair;
2.2 ARCHITECTRRE
PIXEL-base is a 112M parameter ViT-MAE architecture (He et al. 2022) with a 12-layer ViT en- coder (Dosovitskiy al. 202T) and 8-layer Transformer decoder (Vaswani et al. 2017). The encoder has 86M parameters and the decoder has 26M parameters, respectively. The 8-layer de- coder not used for downstream tasks. We give an overview of the architecture below, with more We did not train larger PIXEI. variants for lack of computational resources.
details in Appendix E
Patch Embeddings jected obtain sequence of patch embeddings with 16 16 pixel resolution, to which fixed sinusoidal position embeddings are added
The images produced by the text renderer 2$ are patch-wise linearly pro-
Patch Span Masking Instead of the random masking proce- dure used in VIT-MAE or block-wise masking in BEiT (Bao eet al. (022), PIXEL uses span masking with a 25% masking ratio as outlined in Algorithm []] which masks spans of up to
Algorithm PIXEL Span Masking
Input: #image patches N. masking ratio R. maximumn masked span length S. span length cumulative weights W (w1 .. wrg
S = 6 consecutive image patches with a dynamic number of unmasked patches left between them. The idea behind the span masking approach, inspired by T5 (Raffel et al. 2020) and SpanBERT (Joshi et al., (020a), is that it masks more meaning- ful units of text (full words or phrases) than random masking where the model more often has to fill in (parts of) individual characters, thereby encouraging PIXEL to model a higher level of abstraction. In practice. span masking was slightly more effective than random masking in early prototypes of PIXEL.
Output: Masked patches M . <
repeat
randchoice (l..... .). W)
- rOndint(O, max(O, N - 6))
#f M I+s
1) and
Mn +1....
+8
then
M MU
end if
until [M] > R N return M
This effect may be less noticeable at higher masking ratios (such as the 75% used in ViT-MAF), when random masking would more often masks consecutive patches. We found 25% masking ratio to work well for PiXEL-base, which is in line with recent findings for BERT-type models of simi- lar size (Wettig al. 2022). We mask spans of s E {1, 4} patches in length, each with 20% probability, and spans of s E {5, 6} patches with 10% probability each, so E(s) 3.1.
'We chose sequence length of 529 so that the memory requirements maximum length are approx. equal those of BERT. Forward and hackward passes of the transformer layers at equal length are also equally fast.
"This is fast operation that does not require the large text embedding layer found in llwword-base models, saving parameters which could in theory he re-allocated to the self-attention stack. We refer to Xue (2022 for discussion regarding benefits and drawhacks of re-allocation of emhedding layer weights.
3

---

Published as a conference paper at ICLR 2023
Encoder Following ViT-MAE (He et al.) 2022 the PIXEL encoder only processes unmasked patches (i.e., R 396 "visible" patches at 25% masking) rather than on a sequence including mask tokens, which not only reduces memory requirements and increases training speed, but also has the advantage of not creating a mismatch between pretraining and finetuning. This mismatch would occur when training the encoder with inserted mask tokens because they are not inserted during fine- tuning (He et al. 2022). We also prepend the special CLS embedding to the unmasked patches 66 The resulting CLS and unmasked patches are processed by 12-layer Transformer encoder produce sequence of encoder output representations.
Decoder The PIXEL decoder first projects the encoder outputs into the same space as the decoder model hidden size. It then inserts learnable mask embeddings at the masked positions; these what PIXEL tries to reconstruct at the pixel level. Fixed sinusoidal position embeddings (Vaswani al. 2017) are added to inject order information. After processing this sequence via 8 Transformer layers, a linear projection yields patch logits. Note that the decoder does not have to compute an expensive softmax over subword vocabulary and circumvents the question of whether to tie the subword embedding weights. PIXEL is trained with a normalised mean squared error (MSE) pixel reconstruction loss measuring the discrepancy between normalised target image patches and reconstructed patches. This loss is only computed for masked, non-blank (text) patches.
2.3 PRETRAINING
PIXEL- pase is pretrained on a rendered version of the English Wikipedia and the Bookcorpus (Zhu et al. 2015), which is roughly equivalent to the BERT pretraining data For better compute eff- ciency, we concatenate paragraphs until the maximum sequence length is reached, albeit not across document and book boundaries. Wikipedia has 2B words rendered into 11.4M examples and the Bookcorpus has I.IB words rendered into 5.4M examples; in total ~3.1B words (BERT used 3.3B) rendered into 16.8M examples PIXEL is pretrained for IM steps with batch size 256 (i.c. ~1 epochs) using the AdamW optimizer (Kingma & Bal 2013 Loshchilov linear warmup over the first 50k steps to a peak learning rate of 1.5e-4 and cosine decay to a minimum learning rate of lc Pretraining took 8 days on 8 400BB Nvidia 100 GPUs. We show the loss curve and additional pretraining details in Appendix E We stored PIXEL checkpoints every 10k steps and make them available alongside the fully trained model on the HuggingFace Hub (Wolf) et al. 2020), which we hope will be useful to analyze training dynamics of PIXEL models (Sellam ct al. 2022). Figure in Appendix[B]shrr shows, for three unseen examples, how PIXEL learns to model language over the course of pretraining.
Hutter 2019) with
2.4 FINETUNING
PIXEL can be finetuned for downstream NLP tasks in a similar fashion BERT-liKe encoders by simply replacing the PIXEL decoder with a suitable classification head. By truncating or interpolat- ing the sinusoidal position embeddings, we can finetune with sequences shorter or longer than 529 patches, respectively. The latter, in particular, is common in computer vision applications to finetune on higher resolution images [Touvron et al. 2019 Kolesnikov et al. 20200, Dosovitskiy et al. 2021 He et al. 2022). For most common NLP tasks, we can typically finetune with sequences shorter than 529 to accelerate training while retaining performance. To demonstrate that PIXEL supports a variety of downstream tasks, we conduct finetuning experiments in four settings follows:
Word Classification For word-level tasks like part-of-speech (POS) tagging and named entity recognition (NER), we render each word at the start of a new image patch so that we can create a bijective mapping between words and patches (see Figure for an example) I To finetune PIXEL on these images, we add a linear classifier with dropout. We assign the label a word only to its first corresponding image patch and compute cross-entropy loss with softmax.
Dependency Parsing For dependency parsing, we render text as above but obtain word-level rep- resentations by mean pooling over all corresponding image patches of a word and employ a biaffine parsing head (Dozat& & Manning 2017). following the implementation from Glavaš& & Vulié (2021)
'In pretraining, no loss is computed for the CLS embedding We use a similar Wikipedia dump Devlin et al. (2019) used for BERT (February 1, 2018) and a slightly newer version of the Bookcorpus available at at/httss:// heittss://hugiiggaaat
it can used for finetuning.
*This rendering is quite compact; see Appendix
9This particular formulation assumes that word boundaries available. We note that subword-hard heasord-hased and character-hased models also make this assumption. For further discussion the implications, see Appendix F
4

---

Published as a conference paper at ICLR 2023
Sequence Classification For sequence-level tasks, e.g. in GLUE (Wang et al. 2018), render text as in pretraining. For sentence-paii tasks like natural language inference (NLI) we separate the sentences with black patch. We finetune with different strategies, including training classifier on top of (1) the CLS embedding, (2) the mean-pooled or max-pooled representations of all patches, (3) a multi-head attention block. Although we did not notice significant performance differences between them in our experiments, we mainly used option (I), which is exactly the same as in BERT, and (2), which has been shown work well for image classification (Liang al. 2022).
Extractive Question Answering (QA) For extractive QA datasets like SQuAD (Rajpurkar al., 2016). we render the question and context like in sequence-pair tasks above and, same as Devlin et al. (2019), use a sliding window approach extract answers for examples exceeding the maximum sequence length. We use linear classifier to predict the start and end patches of the span containing the answer. Appendix D explains how we obtain the mapping between characters and rendered text.
3 EXPERIMENTS
We finetune PIXEL on common NLP tasks and evaluate its syntactic and semantic processing capa- bilities in English, as well as its adaptability to unseen languages. languages used in these experiments, and our language and data selection is also motivated below.
describes the
3.1 TASKS AND LANGUAGES
Syntactic Tasks We evaluate PIXEL part-of-speech (POS) tagging and dependency parsing using data from Universal Dependencies 10 treebanks (Nivre et al. 2020; Zeman et al. 2022) for set of typologically diverse languages that captures a large variety of unseen scripts" to Arabic (ARA), Coptic (COP). English (ENG). Hindi (HIN), Japanese (JPN), Korean (KOR), Tamil (TAM), Vietnamese (VIE), Chinese (ZHO) We compare how well PIXEL transfers to these languages compared to BERT. Note that BERT does not support all of these writing systems. However, both models have been trained on the same data. This comparison allows us gauge the extent which PIXEL can overcome the script barrier and vocabulary bottleneck of subword-baas models.
Semantic Tasks We evaluate both monolingual (ENG) and cross-lingual word-level understanding on MasakhaNER (Adelani et al. 2 named entity recognition (NER) benchmark for 10 African languages (AMH, HAU, IBO, KIN, LUG, LUO, PCM, SWA, WOL, YOR), which also includes copy of the ConLL-2003 dataset (ENG; Tjong Kim Sang & De Meulder 2003). For monolingual ENG sentence-level understanding we rely on GLUE (Wang et al., 2018) and SQuAD (Rajpurkar et al. 2016). Finally, we evaluate cross-lingual sentence-leve understanding on TyDiQA-GoldP (Clark et al. 2020) in the in-llanuaage multitask setting where we train on the combined gold data in all 9 target languages (ARA, BEN, ENG, FIN, IND, KOR, RUS, SWA, TEL) at once, and on two additional larger monolingual extractive question answering (QA) corpora: KorQuAD 1.0 (KOR; Lim et al. 2019) and JaQuAD (JPN; So et al. 2022).
3.2 BASELINES AND FINETUNING PROTOCOLS
We compare results to BERT-base which is trained on the same data We do not compare to newer monolingual English models like ROBERTA (Liu et al., 2019) T5 (Raffel et al. 2020) DEBERTA (ee et al. a)) because these models have been pretrained longer on much larger corpora 13 Likewise, we do not compare against models trained on massively multilingual corpora. However, to contextualise the performance of PIXEL in gross-lingue settings, we report results for MBERT and, results are available, for CANINE Clark al. 2022). For BERT, we use the standard finetuning protocols used by Devlin et al. (201)) and the same biaffine classifier for parsing as for PIXEI.. We list finetuning details for all tasks in Appendix
3.3 RESULTS
Syntactic Tasks We present results for POS tagging and dependency parsing in Table [ While BERT is slightly better than PIXEI in the monolingual setting (ENG), PIXEI clearly outperforms
10 By unseen, we mean not present in the pretraining data.
11 Taable 10 in Appendix Flenvissiiiivsss.s gives an overview of the treebanks we use.
We use BERT weights from
wttpg/aaa.....cteaaeeeeeeeeeeeee wt/bert-base-caa
We do not intend to claim state-of-the-art performance, but to demonstrate that PIXEI. can overcome the vocabulary bottleneck and to provide starting point for further research pixel-based encoding of language.
5

---

Published as a conference paper at ICLR 2023
|0]
ENG
ARA
COP
HIN
JPN
KOR
TAM
VIE
ZHO
[UNK]%
Fertility
POS Tagging (Accuracy)
ENG
0
1.2
ARA
1.8
3.7
BERT PIXEL
110M
97.2
95.4
26.5
86.4
87.9
60.0
45.4
84.5
58.6
COP
93.6
1.0
86M
96.7
95.7
96.0
96.3
97.2
94.2
81.0
85.7
92.8
HIN
32.6
2.7
JPN
45.5
1.5
Dependency Parsing (LAS)
KOR
84.7
1.0
BERT
110M
90.6
77.7
13.0
75.9
73.8
30.2
15.2
49,4
28.8
TAM
82.3
1.3
VIE
4.5
2.5
PIXEL
86M
88.7
77.3
83.5
89.2
90.7
78.5
52.6
50.5
73.7
2110
73.2
1.5
Table 1: Results for PIXEL and BERT finetuned for POS tagging and dependency parsing on various Universal Dependencies treebanks. We report test set results averaged over 5 runs each. [0] denotes the number of model parameters. The table on the right shows BERT's proportion of [UNK]S as a measure of (inverse) vocabulary coverage and fertility (i.e., number of subwords per tokenized
word; Acs. 2019
Rust et al. 2021) as a measure of over-segmentation in respective UD treebanks.
HL
[0]
ENG
AMH
HAU
180
KIN
LUG
LUO
PCM
SWA
WOL
YOR
MBERT
104
179M
92.2
0
87.3
85.3
72.6
79.3
73.5
86.4
87.5
62.2
80.0
CANINE-C n-gram*
104
167M
89.8
50.0
88.0
85.0
72.8
79.6
74.2
88.7
83.7
66.5
79.1
CANINE-C#
104
127M
79.8
44.6
76.1
75.6
69.4
63.4
66.6
72.7
60.7
67.9
HERT PIXEL
110M
92.9
0
86.6
83.5
72.0
78.4
73.2
87.0
83.3
62.2
73.8
86M
89.5
47.7
82.4
79.9
64.2
76.5
66.6
78.7
79.8
59.7
70.7
Table 2: Results for PIXEL and BERT finetuned for NER on MasakhaNER. We report test set Fi scores averaged over 5 runs each. BERT outperforms PIXEL in all of the languages that use Latin script, whereas PIXEL does better on AMH, whose script is not covered by BERT's vocabulary. The performance gap is smaller for languages heavier in diacrities, e.g. YOR. It is larger for languages closer to English such as Naija Pidgin (PCM), an English-based creole. #L denotes the number of pretraining languages and * indicates results taken from Clark al. (2022) for additional context.
BERT in the remaining languages. On the lower end, the accuracy gap in favor of PIXEL in ARA and VIE, both languages covered by BERT'S vocabulary, is relatively small (~1%). On the higher end, in COP, where BERT has an out-of-vocaullary (IUNK) token ratio of 93%, the gap is 70% for both tasks. There a strong correlation " between the proportion [UNK]s (shown in Table 1 on the right) and the performance gap. which shows that PIXEL overcomes BERT'S vocabulary bottleneck. These results are further analysed in Appendix
Semantic Tasks We present results for NER in Table 2 for GLUE in Table 3 for QA in Table We also conduct experiments on XNLI in the wranslate-train-all setting which we present in Table 16] in Appendix [] for brevity. We find that BERT consistently achieves higher performance than PIXEL in its pretraining language ENG. Likewise, it often outperforms on languages using the Latin writing system; for instance NER where all languages besides AMII use Latin script, in QA for FIN, IND and SWA. Although BERT has more trainable parameters, this finding indicates that a PIXEL model pretrained for the same number of steps as BERT is slightly worse at semantic tasks, and it may require longer pretraining an additional inductive bias to close the performance gap. Similarly, character-based models also tend to underperform subword-basee 2022), here seen by the CANINE-C results. Since the addition of n-gram embeddings improves the performance of CANINE-C, likely due to boosting entity memorisation capabilities (Clark et al., 2022 we hypothesize that PIXEL may benefit from equivalent enhancements.
models on NER (Keren et al.
For languages where BERT only partially covers the script, such as KOR, JPN and TEL in QA, PIXEL consistently outperforms BERT, sometimes by large amounts (e.g. +63 Fi points better on Ko- IQuAD). In the extreme case where BERT has no coverage of the script whatsoever, seen in NER for AMH, BERT fails completely (0 F1) while PIXEL outperforms the larger, multilingually trained CANINE and performs competitively with its n-gram variant. In other words, PIXEL also overcomes the vocabulary bottleneck of subword-based PLMs in serinn tasks. Note that although BERT was trained on English, its vocabulary has a high coverage of the Arabic script, explaining its good performance in ARA and URD! 15
1*Parsson correlation 0.9, p < 0.001 for POS tagging, 0.95, p 0.0001 for dependency parsing.

---

Published as a conference paper at ICLR 2023
MNLI-M/MM
QQP 364k
QNLI 105k
SST-2
COLA 8.6k
STS-B 5.8k
MRPC 3.7k
RTE
WNLI
[0]
AVG
393k
67k
2.5k
635
110M
84.0 /84.2
87.6
91.0
92.6
60.3
88.8
90.2
69.5
$1.8
80.0
PIXEL
86M
78.1 78.9
84.5
87.8
89.6
38.4
81.1
88.2
60.5
53.8
74.1
Table 3: Results for PIXEI. and BERT finetuned on GLUE. We report validation set performance averaged over 5 runs. The metries are Fi score for QQP and MRPC, Matthew's 's correlation for COLA, Spearman's ' p for STS-B, and accuracy for the remaining datasets. PIXEL achieves non- con-trivial performance scores on GLUE, indicating pixel-based encoders can learn higher-level semantic tasks, but performs worse overall than BERT, so it may require (a) more pretraining steps than subword- tokenized PLMs (b) additional inductive bias to acquire the same level of monolingual abstraction.
TyDiQA GoldP
SQuAD ENG
KorQuAD
JaQuAD
[+]
ENG
ARA
BEN
FIN
IND
KOR
RUS
SWA
TEL
AVG
KOR
JPN
MAERT
104
179M
75.6
78.1
74.7
75.5
84.3
64.8
74.9
83.1
81.6
77.1
88.6
90.0
76.4
BERT
110M
68.5
58.0
43.2
58.3
67.1
124
53.2
51.5
88.2
28.8
PIXEL
86M
59.6
57.3
36.3
57.1
63.6
26.1
50.5
65.9
61.7
52.3
81.4
78.0
34.1
Table 4: Results for PIXEL and BERT finetuned on extractive QA datasets. We report validation set Fi scores averaged over runs each. Average (AVG) scores for TyDiQA-GoldP exclude ENG as customary (Clark et al. 2020). While BERT clearly outperforms PIXEL in ENG, PIXEI is much better in KOR, TEL, and JPN- a consequence of the vocabulary bottleneck in BERT thereby gaining an edge on average. In some languages, answer span extraction adversely affects results (see $3.3).
While the same may apply to languages like BEN and RUS in QA, where one may otherwise expect PIXEL to outperform BERT, there is an external factor at play; in the standard QA task formulation used by BERT, answer spans are extracted by predicting start and end tokens. We adopt this pro- cedure in PIXEL for simplicity. However, an image patch will often overlap two words at variable positions, so the answer may actually start or end mid mid-patch. By only predicting on a full- full-patch level, and extracting the entire content of the patch, PIXEL will sometimes extract leading and trail- ing characters that should not be part of the answer, which degrades the Fl score-even though the model may have correctly identified the span. Languages not using whitespace to delimit words are particularly affected, which also explains why PIXEL is only slightly better than BERT in JPN.
Generally, and in particular when transferring unseen scripts, we find that PIXEL performs best when finetuning on larger corpora. An example of this behaviour can be seen in QA, where PIXEL performs significantly better on KorQuAD (60k examples) than the KOR subset of TyDi (1.6k ex- amples). While large corpora may often not be available when dealing with unseen scripts, we hy- pothesize that multilingual pretraining will alleviate the need for long finetuning. while potentially
being even more conducive to positive transfer (Conneau et al. 2020 Chau et al. 2020 Preiffer
al. 202T by not being vocabulary-bootllnecce. bottlenecked.
4 ROBUSTNESS TO ORTHOGRAPHIC ATTACKS AND
Informal text, commonly found on social media, often contains orthographic noise such as typos and other variations (Baldwin et al. [015]] van Esch ct al.) 2019 Caswell et al.) 2020). Previous work has demonstrated the vulnerability of pretrained language models character- adversarial attacks and noise (Sun et al. 2020 Eger & Benz 2020), with text normalization typically required to main- tain performance (Pruthi et | 2019 Keller Keller et al. 2021). To evaluate PIXEL's 's robustness textual noise and variation and inspired by the robustness tests of Salesky et al. (202T), we experiment with the Zeroé benchmark (Eger& & Benz, 2020 Keller et al. (021) which covers variety of low-level
orthographic attacks as illustrated in Table 13 We replace their version of visual attacks with the We apply Zeroé attacks during finetuning and evaluation of two English downstream tasks, POS tagging and II.I (Bowman et al. 2015). where we expect models to rely on different levels of abstraction.
Unicode Technical Standard #39 set of visually-confusable characters
Arabii is lexically sparse (Antoun et al. 2020 Al-Sallab al. 2017 7. so the characters can be covered in the vocahulary. However, it morphologically complex, which leads to over-segmennation, as the fertility of 3.7 in Table shows. This werr-eegmentation is not necessarily problematic our selection of tasks (Keren ct al. 2022), c.g. due to the sliding window in QA, hut can be a disadvantage in others (Rust et al. 2021
https://uti.. http://uutllu.nideeeeeeee www.codeoogguunnng wttp:///tuuoooeeeps////ooooooo httpsspppssoosssssss...oso
7

---

Published as a conference paper at ICLR 2023
The first Westerner reach Hawaii
Mar black sult, whtre shizt bla
mati ir an sidewal with a guooou of people in backg, Ound, creadig a street. is sweeging.
sWeep sweep/agg
aptain James Cook, the British commande whose mission to disacover the myth
c bowrfe playing an unstruMent with
Hie re of his symphony surrourding h
ic Northwest Passage linking the Atlantic and Pacific oceans. James Cook never went west of California.
lm..
Nobody has a .uit
(a) 0%, contradiction
(b) 80%, contradiction
(c) 80%, entailment
Figure 3: Visual explanations of correct PIXEL predictions (for classes contradiction and entail- ment) for NLI examples with 0% and 80% CONFUSABLE substitutions using method by CChfeer et al. 2021), providing qualitative evidence for PIXEI'ss 's robustness to character-level noise and the interpretability of its predictions. Red heatmap regions represent high relevancy.
Figures 8 and 9 in Appendix G compare PIXEL and BERT across three levels of token-level noise for POS tagging and NLI. There is little impact on POS tagging performance with either model from most low-level attacks, with the exception of visually-confusable character substitutions (CONFUSABLE); here PIXEL expectedly maintains performance above 92% as it generalizes across orthographic similarities but BERT drops to 38%. For NLI, both models are negatively affected, but PIXEL exhibits less degradation than BERT with higher proportions of noise, with the impact varying across the types of attacks which each affect subword tokenization differently. Figure 3 shows rel- evancy heatmaps (Chefer et al.] (02T) for SNLI predictions made with and without CONFUSABLE substitutions. The heatmaps are similarly clear with and without noise, providing qualitative evi- dence that PIXEL is indeed robust to the noise. The illustrated robustness may be dependent upon finetuning. however; find that PIXEL struggle in zero-shot applications when text is rendered differently from observed during pretraining (see Appendix Doon using different fonts). Future work could explore the impact of data augmentation during pretraining on PIXEL'' 's robustness and ability to transfer across scripts. Furthermore, it would be interesting to investigate how the choice of font influences the search space during reconstruction of masked patches [Bland et al. 2022).
In addition to robustness to orthographic noise, deal- ing with character-level substitutions is important for effectively modelling different morphological forms. There are also many types of higher-level token, phrase or sequence-level variations such as code- switching- whhn a speaker alternates between two or more languages in the same utterance, while be-
POS Tagging SPA-ENG HIN-ENG
Named Entity Recognition SPA-ENG HIN-ENG MSA-EA
MBERT
97.1
86.3
64.0
72.6
65.4
BERT
96.9
87.0
74.5
59.4
PIXEI
96.8
88.2
61.0
73.0
63.7
Table 5: Code-switching results on LINCE.
ing grammatically consistent in each language (Joshil 1982) -or the lexical substitutions social media text. We evaluate PIXEL the LinCE benchmark (Aguilar et al. (020)), which includes core tasks and downstream applications for linguistic code-switching. PIXEL is fine-tuned on POS Tag- ging and NER in Spanish-Engie
Hindi Eandi--Engishh and Modern Standard Arabic Arabic-Egyptian Arabic. Tsboe that PIXEL and BERT perform similarly on SPA-ENG tasks, with BERT outperforming PIXEL on NER for (romanised) HIN-ENG. On the other tasks, PIXEL performs better than BERT and even outperforms MBERT on HIN-ENG POS tagging. The gap between MBERT and PIXEL is larger on Arabic scripts, which were extensively seen by MBERT during pretraining.
5 RELATED WORK
The question of vocabulary construction is open problem in NLP, especially in a multilingual context 17 The most widely used language models, e.g. BERT, RoBERTa, T5, GPT-2 inter alia, rely on different tokenizers, such WordPiece (Devlin al. 2019). Byte-Pair Encoding (BPE; [Sennrich] let al. 2016) and Unigram LM (Kudo, 2018). There is an established ecosystem around subword tokenizers, such as the SentencePiece (Kudo & Richardson 2018) and HuggingFace Tokenizers.
In monolingual context and for some languages like English, vocabularies of subwords are good tradeoff between vocabularies of characters and vocabularies of words. When representing large number of languages in multilingual PLMs like mBERT and XLM-R, adequately representing the vocabulary of each individual language would be computationally prohibitive. The tokenization then becomes a bottleneck when trying to scale up to large number of languages (Conneau et al.) 2020 Rust et al. 2021 which manifests itself in degraded cross-lingual performance to languages and
17 See Mielke al. 2021 for a recent, comprehensive survey on open-vocahulary modeling and tokenization.
8

---

Published as a conference paper at ICLR 2023
language families that are underrepresented in the data used for training multilingual PLMs. There are large inequalities in the performance of these models across typologically diverse languages (Wu & Dredze 2020: Lauscher et al. (020). This issue is further exacerbated by tokenizations out-of-
the-box not being compatible across languages (Maronikolakis et al. (021). Language imbalance and poor character coverage in the vocabulary can also decrease downstream performance (Zhang) iet al. (2022). To some extent, these problems can be attenuated through techniques such as subword mapping (Vernikos & Sopescu-Belis 202T), transliteration (Moosa al., 2022), leveraging lexical overlap (Patil et al. (002)) vocabulary clustering and reallocation (Chung et al., 2020), continued or iiguge-adaptive pretraining (Ebrahimi & Kann (021), adaptation via bilingual lexica (Wang et al. 2022), and embedding matrix adaptation (Artetxe et al.) 2020). However, these are post-hoc workarounds to expand model vocabularies after training. They do not provide a direct solution to the vocabulary bottleneck problem.
Some subword-based algorithms can also produce undesirable segmentations for morphologically rich languages (Klein & Tsarfaty 2020 Amrhein & Sennrich 202I)) so dedicated morphologically- aware tokenizers have been developed (e.g. Smii et al. (2014))))) but this process often requires expert- level knowledge and may only work for individual languages.
Due to the limitations of subword vocabularies in multilingual language modelling. some works have used vocabularies over characters (Lee et al.] 2017] Ma et al. 2020 inter alia) or bytes (Wang) et al. 2020; Wei et al. 202T). These provide benefits over purely subword-based models in terms of robustness and most of them are readily applicable in multilingual context [8] but they typi- cally come at the cost of increased sequence lengths or latency. Also, such models cannot exploit orthographic similarities between characters across and within scripts and do not account for the fact that meaning of language may be carried visually such as in writing systems that are (partially) logographic like Chinese, in ancient hieroglyphs, or when using emoji.
Finally, some works have developed pixel-based approaches. Broscheit (018) embedded images of Chinese glyphs but still relied on fixed vocabulary. |Wu et al. (2019) combined character-level images and embeddings for a variety of Chinese tasks. Radford et al. (202T) trained a linear probe for CLIP, which also incorporates tokenizer, on a rendered version SST-2 (Socher et al.) 2013). Other works have trained pixel-based models that removed the need for fixed vocabulary: (Sun et al. 2019) trained a convolutional sentiment classifier on pixels. Mansimov et al. (2020) used images of text for in-image MT. Salesky al. (202I) employed a convolutional embedder for Transformer- based MT system with subword- basedd decoder. Our method differs from these in that it provides general-purpose language encoder that completely removes the need for vocabulary.
6 CONCLUSION
This paper introduced PIXEL, pretrained language model that renders text as images, which allows it to represent any written language that can be typeset using its text renderer. PIXEL was pretrained on the predominantly English Wikipedia and Bookcorpus datasets, and evaluated on part-of-speech tagging, dependency parsing, question answering, and language understanding tasks. The results demonstrate that PIXEL readily transfers to unseen scripts, as shown by its performance on 14 scripts across 32 languages. PIXEI. currently lags behind BERT when processing languages with a Latin script, including English; however, PIXEL more robust than BERT against low-level orthographic attacks and performs competitively to BERT and MBERT linguistic code-switching tasks. Overall, these results show that pixel-based representations are a strong backbone for longull and cross- script transfer learning. The limitations of this work are discussed in Appendix []
In future work, we will investigate inductive biases and additional objectives that can better capture range dependencies in PIXEL models. We hope that this will help overcome the limits of PIXEL in semantic processing. We also plan to pretrain PIXEL on multilingual text with a view to further improving its cross-script and cross-lingual abilities. This will also allow fairly compare pixel-based models against larger subword-basee and tokenization- mudtilingual models. Finally, we will also develop new rendering and finetuning formulations that are better tailored to pixel-based models, e.g. for improving downstream question answering.
more
"CCharacter-wware models are not directly applicable to languages that do not use whitespace to delimit sentences (Tay et al. 202T), for example.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Ákos Kádár, Barbara Plank, and Kris Cao for their comments on an earlier draft. We also thank Davide Rigoni, Rita Ramos, Stella Frank, and members of the CoAStaL and LAMP groups for discussions. Miryam de Lhoneux is funded by the Swedish Research Council (grant 2020-00437). Phillip Rust is funded by the Novo Nordisk Foundation (grant NNF 20060665)... Jonas F. Lotz is funded by the ROCKWOOL Foundation (grant 1242). Emanuele Bugliarello is supported by funding from the European Union's 's Horizon 2020 research and innovation programme under the Marie Sktodowska-Curie grant agreement No 801 199. Elizabeth Salesky is supported by the Apple Scholars AI/MI fellowship. Desmond Elliott is partially supported by the Innovation Foundation (grant 01 76-60001133 and the Novo Nordisk Foundation (grant NNF 208A0066568.. This work was supported by a research grant (VIL53122) 122) from VILLUM FONDEN. The computing power was generously supported by EuroHPC grants 2010PA5869, 2021D02-068 D02-068, and 2021 141, and with Cloud TPUs from Google's TPU Research Cloud (TRC).
REFERENCES
Judit Ács. Exploring BERT's Vocabulary. Blog Post, 2019. URL http://juditacs tac.. wwwaacc...ggiiii github.... io/ 20119//779/brrt--ooeeezztoooosss
David Ifcoluwa Adelani. Jade Abbott, Graham Neubig, Daniel D' D'souza, Julia Kreutzer, Constan- tine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende: Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Je- sujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo An- dre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson. Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chia- maka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, To- bius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. MasakhaNER: Named Entity Recog- nition for African Languages. Transactions of the Association for Computational Lingnis- fics, 9:1116-1131, 2021. ISSN 2307-387X. doi: 10.1162/ a_406... URL. https: //doi.org/10. 1162/////1..1//alll 00416
Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. LinCE: Centralized Benchmark for Linguis- tic Code-switching Evaluation. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 1803-1813, Marseille, France, May 2020. European Language Resources Asso- ciation. ISBN 979-10-95546-344 URL hhttpp:://wwt aclweb....ww..cooo.............b hwwp:/ewe.rrrr///eeeeeee htgg//nnthology2222.ooo Irec-
1.223
Ahmad AI-Sallab, Ramy Baly, Hazem Hajj. Khaled Bashir Shaban, Wassim El-Hajj. and Gilbert Badaro. Aroma: A recursive deep learning model for opinion mining in arabic low resource language. ACM Trans. Asian Low-Resour Lang. Inf. Process. 16(4), jul 2017. ISSN 2375-4699.
doi: 10.1145/3086575 URL/https /tttps://doi.ooo org / 1115/000111/338855555
Chantal Amrhein and Rico Sennrich. How suitable are subword segmentation strategies for translating non-concatenative morphology? Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 689-705, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202..mmm www.diingnggeeell....... URL https://aclanntholoy..oom wwwpthhoologo.oo/2///// 2021 findings-emnip.ee
Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with Shared Task Offensive Language Detection, pp. 9-15, Marseille, France, May 2020. European Language Resource Association. ISBN 979-10-95546-51- URL https://acl.anthoogyy.orrg osact-1.2
10

---

Published as a conference paper at ICLR 2023
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross- lons-lingual transferability of mono- lingual representations. In Proceedings of the 58th Anmual Meeting of the Association for Com- putational Linguistics, pp. 4623 0623-4637, Online, July 2020. Association for Computational Lin- URL httpsss/aaclaantooooo..r////////
guistics. doi: 10.18653/1 12155///112222al--mama
acl-
Masayuki Asahara, Hiroshi Kanayama, Takaaki Tanaka, Yusuke Miyao, Sumire Uematsu, Shinsuke Mori, Yuji Matsumoto, Mai Omura, and Yugo Murawaki. Universal Dependencies version 2 for Japanese. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- tion (ELRA). URL htwp:: https://aaaaanthology/ orgoorg//// /0gg/L1--122 1287
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint, 2016. URL http::////aa httt:://arxv..rr/aab///6/// 06450
Timothy Baldwin, Marie Catherine de Marneffe, Bo Han, Young-Bun Bum Kim, Alan Ritter, and Wei Xu. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normal- ization and named entity recognition. In Proceedings of the Workshop on Noisy Uxer-generated Text, pp. 135, Beijing, China, July 2015. Association for Computational Linguisties. doi: 10.18653/v 1/W15-44119 URL httttts:/aclaathlllllt or
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEIT: BERT pre-training of image trans- formers. In International Conference on Learning Representations, 2022. URL https://
@penreview. wwt.:nt/oorrmiidp--hh/5SS-
Maxwell Troy Bland, Anushya lyer, and Kirill Levchenko. Story beyond the eye: Glyph positions break PDF text redaction. arXiv preprint, 2022.
Terra Blevins and Luke Zettlemoyer. Language contamination explains the hww.ppppiiiiiiooooooooooooaoaaaa of english pretrained models. arXiv preprint, 2022. URL http://s///o...rr/////////////// 2204. 08110
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models. arXiv preprint, 2022. URL https://aaxii: org/ abs/2205. 11758
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large an- notated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.1865//1/DD15-107. URL https://aclantholoogy 001p://000....//01-11101
Samuel Broscheit. Learning distributional token representations from visual features. In Proceed- ings of The Third Workshop Representation Learning for NLP, pp. 187-194, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. doi: 10. 1/01//301111103 URL https://aclanthollog. wrt.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Sys- tems, volume 33, pp. 1807-1901. Curran Associates, Inc., 2020. URL http:/ttts:/prrceennnngg neurips. CC/ httpcc//ppppeee. 2tapper/222//10 (t1e22/fflle httpaare222////eee////ccoooaaaa0
Emanuele Bugliarello. Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti. and Ivan Vulié. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Proceedings of the 39th International Conference on Machine Learning, Balitmore, MA, July 2022. PMLR. http:ts:///aaii..oo///////////// 11732

---

Published as a conference paper at ICLR 2023
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language id in the wild: Un- expected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference (Online), 2020. Association for Computational Linguistics. URI. hwtp://olllatttooooo.o.ooo org/ 2020. coling main. (tl.n.579.... 579. .ddf
Computational Linguistics, pp.
Barcelona, Spain
Ethan C. Chau, Lucy Lin, and Noah A. Smith. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1324-1334, Online, November 2020. Association for Computational Lin- guisties. doi: 10. 10.18653/112020.nnnniii wwn.:iiinnnnn--el.111... htthtt www.htttss/aaalannoolooo org/ findings-emnlp. 118
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-mode explainability for inter- preting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV), pp. 397-406, October 2021. URL https:://penacceess. theccf. com/ httee:t.cccccotteet/////////////
BaaeraaioMddaaaaa__n_eeeeaee _and -nd_EEccder-Decodrr
Jayeol Chun, Na-Rac Han, Jena D. Hwang, and Jinho D. Choi. Building Universal Dependency treebanks in Korean. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Re-
sources Association (ELRA). URL htttsss//allannhoooe..orrr
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference Em- pirical Methods Natural Language Processing (EMNLP), pp. 4536-4546, Online, November 2020. Association for Computational Linguistics. doi: 10. 10116653/1122220.mmmp-maa https://aclanthologr orwpiaanttolooy.rgg//e222 www.orggg////// wttp://222...mmpmmaaaa... emnlp-main.33 367
URL
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics. 8:454 470, 2020. doi: 10. 162/tacl 00317. URL httpsh::/aaaaaathllllaa org. 1.30
.rg 2020. tacl-
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. Canine: Pre-training efficient tokenization-free encoder for language representation. Trans. Assoc. Comput. Linguistics, 10: 73-91, 2022. doi: 10.11622tacll al 00448. URL http:https://doo
1162111/aacclaaa0000 __0_a______048_____1___ 00448
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross- Pross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, October-Novembor 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/DI.. 1269.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w1/2220.cm maid. 747. /acclanhhooogy, org 0000..000..20 acl-main. 747
URL https:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171 1-7164,86, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19. 1423. URL https: www.clannholoo...rggg/iieo
12

---

Published as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. International Conference on Learning Representations, 2021. URL https: wwtpre/wwww.ee/fooommmdd-eeeeeee
Timothy Dozat and Christopher D. Manning. Deep biaffine attention for neural dependency parsing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
//openreview. http//oorummmi-HH9PKKK.
Abteen Ebrahimi and Katharina Kann. How to adapt your pretrained multilingual model to 1600 languages. In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4555-4567, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022
long.331. URL htt.:/ttps//aa.aatoooooo...e or0/ acl- 1ong-loog.333 351
2021
Steffen Eger and Yannik Benz. From hero to zéroe: A benchmark of low-level adversarial attacks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computa- tional Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 786 803, Suzhou, China, December 2020. Association for Computational Linguisties. URL www.anoooooooogo///2222222 2020 aacl-main.7
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, 2020. URL httts:://aaxiii http://aaxxvvooogaaaaaaa
Goran Glavaš and Ivan Vulié. Is supervised syntactic parsing beneficial for language understanding tasks? an empirical investigation. In Proceedings the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3090-3104 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 158653/v1/202.eerr 1aro21.eacl--ae mallmain.227. URL
https://aclanthoology org/ eacl-main. 270
Jan Hajič, Otakar Smrž, Petr Zemánek, Petr Pajas, Jan Šnaidauf, Emanuel Beška, Jakub Kraemar, and Kamila Hassamová. Prague arabic dependency treebank 1.0, 2009. URL https:://faa. ff http:i/c//paaa/PD..... 0/ bedocs//ndee. html
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Gir- shick. Masked autoencoders
scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, http://m/.ccomeeet/////////////p Maskee
2022. URL https::/openaccess: htw.r/eessttheec.c.cooo/eeeeeeee Autoencoders_ Brrroddrrssrree_eee httpgeresa..aallllliloioeeeeeerr
.pdf]
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangle embedding sharing. arXiv preprint, 2021a. URL
09543
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations, 2021b.
URL
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint, 2016. URL http:://arxiv..rrgabb//////////s 10tpg///aaa/11668888855555 08415
Julia Hirschberg and Christopher D Manning. Advances in natural language processing. Science, 349(6245):66122666 2015. UtL/htttss:/ccs224d.. stwnford...roorrrrooooooooo wttp:///wdd///pppppaa/aaaaeeeeee
Aravind K. Joshi. Processing of sentences with intra-sentential code-switching. In Coling 1982: Proceedings of the Ninth International Conference on Computational Linguistics, 1982. URL
https:s//aclaattoooyy.orr//////
13

---

Published as a conference paper at ICLR 2023
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the As- sociation for Computational Linguistics, 8:64-77, 2020a. doi: 10.1162/tacl a 00300. URL
hwwp:/ooggooog/////// org/2020. tacl-1.5
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online, July 2020b. https:
Association for Computational Linguistics. doi: 10.18653/w1/202..cz mm..mm..55. //aclanthology. org/ 2020. acl-main. 560
Yannik Keller, Jan Mackensen, and Steffen Eger. BERT-defense: A probabilistic model based on BERT combat cognitively inspired orthographic adversarial attacks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1616-1629, Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021. http://w/220...nnnnnnaell..... URL.
https://acllanthologg..uo org/2221 weeie.finndnggg "indingss-cll 141
Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. Breaking character: Are subwords good enough for mrls after all? arXiv preprint, 2022.
.47448
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, USA, 2015. URL http://///axx http:///aaxxiv.ooraaa////ss/ hrtp:/s////////////// 6980
Stav Klein and Reut Tsarfaty. Getting the ##lifee out of living: How adequate are word-pieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop Computational Research in Phonetics, Phonology, and Morphology, pp. 204-209, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18663/11/0020sigmmrrr. 1.24. URL httpp:://alaanthology
http://awnnnnnnooooor/ror//////e www.orgg//2/// sigmorphon-l11111
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 491-507, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58558-7, URL 1007/978 000000000088--8-
Taku Kudo. Subword regularization: Improving neural network translation models with multi- ple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1000 s/aclanthology.org//TP org/
URL https:
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018 Association for Computational Linguistics. doi: 10. 10133/11/D1-20111 2012.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 4483- 4499, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ www.emmllmmiii336666 URL htttps://aclanttollgy/// wwwpgy.orgg/200 emnlp-main 363
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine translation
without explicit segmentation. Transactions of the Association for Computational Linguistics, 5: 365-378, 2017. doi: 10. 11.11/2/ a_ a006667. URL httlltttss//cllannooo.oe.e wtwp://oowwyooo/111111110101
Feng Liang, Yangguang Li, and Diana Marculescu. Supmae: Supervised masked autoencoders efficient vision learners. arXiv preprint, 2022. htllitttss//arxivorr////// 2 14540
Seungyoung Lim, Myungji Kim, and Jooyoul Lec. Korquad1.0: Korean QA dataset for machine reading comprehension. arXiv preprint, 2019. httphhttpp//aaxx..ogg/ab////////
14

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint, 2019. http:/t///aaxii.orrr
/ttp//////9/.00 11692
Iyy Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL prttps://openreeet net/
forum?id=Skg895oxx
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 2019. OpenReview.net. URL
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. CharBERT: Character-aware pre-trained language model. In Proceedings of the 28th International Con- ference on Computational Linguistics, pp. 39-50, Barcelona, Spain (Online), December 2020, International Committee Computational Linguisties. doi: 10.18653112020.ccliim main.. URL httpp::/aclanthhllogg wttp:/oorrr022.....ogooaaaaaaaaa
Elman Mansimov, Mitchell Stern, Mia Chen, Orhan Firat, Jakob Uszkoreit, and Puneet Jain. To- wards end-to-end in-image neural machine translation. In Proceedings of the First Interna- tional Workshop on Natural Language Processing Beyond Text, pp. 70-74, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/20220 122003//1/02....pbb1111.... URL
https::/aac.amtioooogy.o//////// nlpbt-1.8
Antonis Maronikolakis, Philipp Dufter, and Hinrich Schütze. Wine is not i on the compatibil- ity of tokenizations across languages. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pp. 2382-2399, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: https:://aclanttoologyy org.oog///////// 2021. findings-emninn 205
URL
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Samson Tan. Between words and characters: A history of open- vocabuluyy modeling and tokenization in NLP. arXiv preprint, 2021. URL
https://arxiv. www.ooggg//.../... /tw.ab///..... 10508
Ibraheem Muhammad Moosa, Mahmud Elahi Akhter, and Ashfia Binte Habib. Does transliteration help multilingual language modeling? arXiv preprint, 2022. URL https://arxvv.or/////
2201 12501
Phuong- Thai Nguyen, Xuan-Luong Vu, Thi-Minh-Huyen Nguyen, Van-Hiep Nguyen, and Hong-
Phuong Le. Building a large syntacticall--annotaeee corpus of Vietnamese. Proceedings of the
Third Linguistic Annotation Workshop (LAW III), pp. 185, Suntec, Singapore. August 2009. 3035
Association for Computational Linguistics. URL https: http://acaaanooooo.rrrrg////////
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo. Sebastian Schuster, Francis Tyers, and Daniel Zeman. Universal Dependen- cies v2: An evergrowing multilingual treebank collection. Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4034 4043, Marseille, France, May 2020. European Language Resources Association. ISBN 17-9-0095554--35 34-4. URL https: https:://ccaaaa haat:::://allanhhlooyyy org/2020. 1rec-1 .497
Martha Palmer, Owen Rambow, Rajesh Bhatt, Dipti Misra Sharma, Bhuvana Narasimhan, and F. Xia. Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In Proceedings ICON-2009: 7th International Conference on Natural Language Pro- cessing, India, 2009. Macmillan Publishers. URL htttt:::/ccm// iiit.ac. wwt/nn//t/nn.iiiiiiii/ic iiit. ac. www.rreeereeateeepuulllieeee
.pdf]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chil- amkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An impera- tive style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle,
15

---

Published as a conference paper at ICLR 2023
Alina Beygelzimer, Florence d' d'Alché-But Buc, Emily B. Fox. and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver. BC, Canada, pp. 8024-8035, 2019.
Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. Overlap- vocabulary generation improves cross-lingual transfer among related languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 219-233, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 20.118553v1//// /ttp://////2222alllloooo acl-long. 18
18. URL
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. UNKs everywhere: Adapting multilingual language models new scripts. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 10186-10205, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011. org/ /2wg./2211.222 www.p-main.nnemmmmmmmmmmmmmmm.em 800
emnlp-main.800 800. URL
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: URL
10.18653/v1/P19. 1493.
Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Anmual Meeting of the Association for Com- putational Linguistics, pp. 5582-5591, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/PP 1/165///1/11915550 URL httpp::/acclanthologg/ 11tpaa//oooooooooo////995115
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. $748-8763, PMLR, 2021. URL http:///proceedings mlr, mll.press/r/rp/// M139/radford2la.htaa
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text- to-text transformer. Journal of Machine Learning Research, 21(140):1-67 2020. URL http:
haimm//.orrgpppps/v//// 01mmmmmmm
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 2883-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 16..18533v1/D16-1110 1264. URL httpps://a.lanthllllo org/ D16-1264
Loganathan Ramasamy and Zdeněk Žabokrtsky. Prague dependency style treebank for Tamil. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC' 12), pp. 1888-1894, Istanbul, Turkey, May 2012. European Language Resources Associ- _aper__
ation (ELRA). URL http::/www. lrec- conf www.ooooo/oooooooooooooooooo http:///rrooeeinn/////////////// idf
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pa- pers), pp. 3118-3133 Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 http:////2222..112..2..22222 a43--log..243. URL htttt:::s//caattthhhh http///aaaaaanttooaooo/o hrw.:llogo.oooo//a//ooeoo wrw.oog////20 2221 acl-long.22 243
Elizabeth Salesky, David Etter, and Matt Post. Robust open-vocabulat translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural
16

---

Published as a conference paper at ICLR 2023
Language Processing, pp. 7235-7252, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653//1/2021.eem main.576. URL
fttpss//aalancchoogyy.oo//////// emnlp-mainn 576
Thibault Sellam, Steve Yadlowsky, lan Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Fisenstein, Dipanjan Das, and Ellie Pavlick. The multiBERTs: BERT reproductions for robustness analysis. In International Conference on Learning Representations, 2022. URL https:://opeerreiewww http://wwwweeeeoooom////eeee_//
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P1-111 URL/htpss:/aclantho .rg//
P16-1162
Mo Shen, Ryan McDonald, Daniel Daniel Zeman, and Peng Qi. Ud chinese-gsd. GitHub repository, 2016.
URL https://gtt
Natalia Silveira, Timothy Dozat, Cerie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC") 14), pp- 2897-2904, Reykjavik, Iceland. May 2014. European Language Resources Association (ELRA). URL http://www wrec- conf org httpp//eeeeninneeeeeee/////////e Pap pdf
Peter Smit, Sami Virpioja, Stig-Arne Grönroos, and Mikko Kurimo. Morfessor 2.0: Toolkit for statistical morphological segmentation. In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp- 21-24, Gothenburg, Sweden, April 2014. Association for Computational Linguistics. doi: 1014-0000 URL htthhttts///aaaantologg wwwpaanoooooooooo//1-/-220000000 org/E14-2006
ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. JaQuAD: Japanese question
answering dataset for machine reading comprehension. ArXiv preprint, 2022. URL https:
//arxivoor//as//202.000 01764
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL httphhttts//acaatthhtttoo
Baohua Sun, Lin Yang, Catherine Chi, Wenhan Zhang, and Michael Lin. Squared english word: A method of generating glyph to super characters for sentiment analysis. In AffCon@AAAl 2019. URL phttps:://arxtt http:///aaxxw..o////////////////
www.grg////////////ee/ee// 02160
Lichao Sun, Kazuma Hashimoto. Wenpeng Yin, Akari Asai, Jia Li, Philip S. Yu, and Caiming Xiong. Ady-bert: BERT not robust on misspellings! generating nature adversarial samples on 04985
BERT. arXiv preprint, 2020.
Yi Tay, Vinh Q Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Si- mon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, 2021. URL http//tttt///eeeeeieiiiee wtt.://wwww.eeeoooooddddddaaaaa 10EFN
Owen Taylor. Pango, open-source unicode text layout engine. In Proceedings of the 25th Interna- tionalization and Unicode Conference, Washington, D.C., USA, 2004. The Unicode Consortium. pdff]
URL https:://ppppple redhat. com/ http:///.ooooooooooooooooo/o///t
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-indeeenderr named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https://
acllnthology.oot w0wg/////00019999
17

---

Published as a conference paper at ICLR 2023
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test reso- lution discrepancy. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL.
neurips http:/iiww..eerrrs...cc/p/////// cc/paper/2019/ffll/
Iulia Ture, Kenton Lee, Jacob Eisenstein, Ming-Wes Chang, and Kristina Toutanova. Revisiting the primacy of english in zero-shot cross-lingual transfer. arXiv preprint, 2021. URL https:
//arxiv.org/abs//110 16171
Daan van Esch, Elnaz Sarbar, Tamar Lucassen, Jeremy O' O'Brien, Theresa Breiner, Manasa Prasad, Evan Crew, Chieu Nguyen, and Françoise Beaufays. Writing across the world's languages: Deep internationalizzaionn for gboard, the google keyboard. Technical report, 2019. URI. http:// brxiv.org/abs//990 01218
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL https: httttp:://proceedinggs. wewpeeeddiinns.uuuussss.... wwtppppspcccppppppppeppppppeeept /217/has2/ 3f5ee243547 c
Giorgos Vernikos and Andrei Popescu-Belis Subword mapping and anchoring across languages. In Findings of the Association for Computational Linguisties: EMNLP 2021, pp. 2633-2641 Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 18.118811/222111
http://////2....innneeemmmm///// findinngs-emllp2224
URL https://acllatthllooe /ttpaaatooooooor/ooo/.... 2021.
findings-emnlp. 224
Ada Wan. Fairness in representation for multilingual NLP: Insights from controlled experiments on
conditional language modeling. In International Conference on Learning Representations, 2022.
URL httptss:/ppeeerreiieeee wtwprrrewwweeeeeorunn/.e... net/forum?id=-1a 1ewriid===11161000 1156Ti0ew
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pp. 353 355. Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10.18653/v 1/t1 1-WW88454..... URL
8 --446
Changhan Wang, Kyunghyun Cho, and Jiatao Gu. Neural machine translation with byte-level sub- words. In The Thirty-Fourte Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9154-9160. AAAI Press, 2020. URL https:://ojs. aaai org/index. hhpp/MAl/article//iew/w// 6451
Xinyi Wang, Sebastian Ruder, and Graham Neubig. Expanding pretrained models to thousands more languages via lexicon-based adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 863-877, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 20.11865//11/0000 001/33/1122222.o lon- 61. http://ppss/aaaanhtllleeoooo wrgp:/yy.rrg222222/ acl long.
Junqiu Wei, Qun Liu, Yinpeng Guo, and Xin Jiang. Training multilingual pre-trained language model with byte-level subwords. arXiv preprint, 2021. URI. hhttps://arxii wrr.vi.orgg/bbao a111 09469
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? arXiv preprint, 2022. httpp::///// http://awwiiv.oogbb/////s/o/ wttp://///a//......./g
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemntee Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger,
18

---

Published as a conference paper at ICLR 2023
Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: oftte----hhe-rr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Asso- ciation for Computational Linguistics. doi: 1011665//122200mmmmmdddmm TTaclanttology.orrg//2e2 emw.p-deesp--dd
URL https:
Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT? In Pro- ceedings of the 5th Workshop Representation Learning for NLP, pp. 120-130, Online, July 2020. Association for Computational Linguistics. doi: 1011865//1100000xp/hhhppll https://aclanthollogy
URL
Wei Wu, Yuxian Meng. Fei Wang, Qinghong Han, Muyu Li, Xiaoya Li, Jie Mei, Ping Nie, Xi- aofei Sun, and Jiwei Li. Glyce: Glyph-vectors for chinese character representations. In Neural Information Processing Systems, 2019.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10:291-306, 03 2022. ISSN 2307-387X.
doi: 1 00461. URL https::/doo..rrg///// 11l.1//.111/taacc..
00461
Amir Zeldes and Mitchell Abrams. The Coptic Universal Dependency treebank. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pp. 192-201, Brussels, Belgium, November 2018. Association for Computational Linguisties. doi: 10. N0.18653/11/11-666 URL
https://aclanthol.ogg -
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Zeljko Agié, Amir Ahmadi, Lars Ahrenberg, Chika Kennedy Ajede, Gabrielé Aleksandravičintéé Ika Alfina, Avner Algom, Erik Andersen, Lene Antonsen, Katya Aplonova, Angelina Aquino, Carolina Aragon, Glyd Aranes, Maria Jesus Aranzabe, Bilge Nas Arcca, Hórumn Arnardóttrr, Gashaw Arutie, Jessica Naraiswari Arwidarasti, Masayuki Asahara, Deniz Baran Aslan, Cen- giz Asmazoğlu, Luma Ateyah, Furkan Atmaca, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Starkadur Barkarson, Rodolfo Basile, Victoria Bas- mov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Kepa Bengoetxea, Yifat Ben Moshe, Gözde Berk, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agné Bielinskiené, Kristín Bjarnadóttir, Rogier Blokland, Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Anouck Braggaar, Kristina Brokaité, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Lauren Cassidy, Tatiana Cavalcanti, Gülşen Cebiroğlu Eryigit, Flavio Massim- iliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Neslihan Cesur, Savas Cetin, Özlem Çetinoglu, Fabricio Chalub, Shweta Chauhan, Ethan Chi, Taishi Chika, Yongseok Cho, Jinho Choi, Jayeol Chun, Juyeon Chung, Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Cağn Çöltekii, Miriam Connor, Daniela Corbetta, Marine Courtin, Mihacla Cristescu, Philemon Daniel, Elizabeth Davidson. Mathieu Dehouck, Martina de Laurentis, Marie-Catherine de Marn- effe, Valeria de Paiva, Mehmet Oguz Derin, Elvis de Souza, Arantza Diaz de Ilarazza, Carly Dickerson, Arawinda Dinakaramani, Elisa Di Nuovo, Bamba Dione, Peter Dirix, Kaja Dobro- volje, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Hanne Eckhoff, Sandra Eiche, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomaž Erjavec, Aline Etienne, Wograine Evelyn, Sidney Facundes, Richard Farkas, Federica Favero, Jannatul Ferdaousi, Marília Fernanda, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarina Gajdošová, Daniel Galbraith, Federica Gamba, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Fabrício Fer- raz Gerardi, Kim Gerdes, Filip Ginter, Gustavo Godoy, lakes Goenaga, Koldo Gojenola, Mem- duh Gökurmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta González Saavedra, Bernadeta Griciūte, Matias Grioni, Loic Grobol, Normunds Grüzïtis, Bruno Guillaume, Céline Guillot- Barbance, Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Hajič, Jan Hajič jr., Mika Hämäläänen, Linh Ha My. Na-Rae Han, Muhammad Yudistira Hanifmuti, Takahiro Harada, Sam Hardwick, Kim Harris, Dag Haug. Johannes Heinecke. Oliver Hellwig, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Jena Hwang. Takumi Ikeda, An- ton Karl Ingason, Radu Ion, Elena Irimia, Olájídé Ishola, Kaoru Ito, Siratun Jannat, Tomáš
19

---

Published as a conference paper at ICLR 2023
Jelínek, Apoorva Jha, Anders Johannsen, Hildur Jónsdóttir, Fredrik Jorgensen, Markus Juuti- nen, Sarveswaran K, Hüner Kaşıkara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hi- roshi Kanayama, Jenna Kanerva, Neslihan Kara, Ritván Karahóğa, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerová, Jesse Kirchner, Elena Klementieva, Elena Klyachko, Arne Köhn, Abdullatif Köksal, Kamil Kopacewiez, Timo Korkiakangas, Mehmet Köse, Natalia Kot- syba, Jolanta Kovalevskaite, Simon Krek, Parameswari Krishnamurthy, Sandra Kübler, Oguzhan Kuyrukçu, Ash Kuzgun, Sookyoung Kwak, Veronika Laippala, Lucia Lam, Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phucngg LÅ Hông. Alessan- dro Lenci, Saran Lertpradit, Herman Leung, Maria Levina, Cheuk Ying Li, Josie Li, Keying Li, Yuan Li, KyungTae Lim, Bruna Lima Padovani, Krister Lindén, Nikola LjubeKić, Olga Loginova, Stefano Lusito, Andry Luthfi, Mikko Luukko. Olga Lyashevskaya, Teresa Lynn, Vivien Macke- tanz, Menel Mahamdi, Jean Maillard, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Büşra Marşan, Cätälina Märänduc, David Mareček, Katrin Marheinecke, Stella Markantonatou, Héctor Martínez Alonso, Lorena Martín Rodríguez, André Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Alessandro Mazzei, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça. Tatiana Merzhevich, Niko Miekka, Karina Mischenkova, Margarita Misir- pashayeva, Anna Missilä, Cätälin Mititelu, Maria Mitrofan, Yusuke Miyao, AmirHossein Mo- jiri Foroushani, Judit Molnár, Amirsaeid Moloodi, Simonetta Montemagni. Amir More, Laura Moreno Romero, Giovanni Moretti, Keiko Sophie Mori, Shinsuke Mori, Tomohiko Morioka, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Mariam Nakhlé, Juan Ignacio Navarro Horñiacek, Anna Nedoluzhko, Gunta Nešpore-Berzkalne, Manuela Nevaci, Luông Nguyên Thi, Huyên Nguyên Thi Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adédayō Olúökun, Mai Omura, Emeka Onwuegbuzia, Noam Ordan, Petya Osenova. Robert Östling, Lilja Øvrelid, Şaziye Betül Özateş. Merve Özşelik, Arzu- can Özgür, Balkız Öztürk Başaran, Teresa Paccosi, Alessio Palmero Aprosio. Hyunji Hayley Park, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme Paulino- Passos, Giulia Pedonese, Angelika Peljak-Lapinskk, Siyao Peng, Cenel-Augusto Perez, Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Andrea Peverelli, Jason Phelan, Jussi Piitu- lainen, Tommi A Pirinen, Emily Pitler, Barbara Plank, Thierry Poibeau, Larisa Ponomareva, Martin Popel, Lauma Pretkalnipa, Sophie Prévost, Prokopis Prokopidis, Adam Przepiórkowski, Tiina Puolakainen, Sampo Pyysalo, Peng Qi, Andriela Rääbis, Alexandre Rademaker, Miza- nur Rahoman, Taraka Rama, Loganathan Ramasamy, Carlos Ramisch, Fam Rashel, Moham- mad Sadegh Rasooli, Vinit Ravishankar, Livy Real, Petru Rebeja, Siva Reddy, Mathilde Reg- nault, Georg Rehm, Ivan Riabov, Michael Rießler, Erika Rimkuté, Larissa Rinaldi, Laura Rit- uma, Putri Rizqiyah, Luisa Rocha, Eiríkur Rögnvaldsson, Mykhailo Romanenko, Rudolf Rosa, Valentin Rosca, Davide Rovati, Ben Rozonoyer, Olga Rudina, Jack Rueter, Kristján Rúnarsson, Shoval Sadde, Pegah Safari, Benoît Sagot, Aleksi Sahala, Shadi Saleh, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Ezgi Samyar, Dage Särg. Baiba Saulīte, Yanin Sawanakunanon, Shefali Saxena, Kevin Scannell, Salvatore Scarlata, Nathan Schneider, Sehastian Schuster, Lane Schwartz, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Syeda Shahzadi, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Yana Shishkina, Muh Shohibussirri, Dmitry Sichinava, Janine Siewert, Einar Freyr SigurOsson, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó. Mária Šimková, Kiril Simov, Maria Skachedubova, Aaron Smith, Isabela Soares-Bastos, Shafi Sourov, Carolyn Spadine, Rachele Sprugnoli, Vi- vian Stamou, Steinhór Steingrímsson, Antonio Stella, Milan Straka, Emmett Strickland, Jana Str- nadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Daniel Swanson, Zsolt Szántó, Chihiro Taguchi, Dima Taji, Yuta Takahashi, Fabio Tamburini, Mary Ann C. Tan, Takaaki Tanaka, Dipta Tanaya, Mirko Tavoni, Samson Tella, Isabelle Tellier, Marinella Testori, Guillaume Thomas, Sara Tonelli, Lissi Torga, Marsida Toska, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sumire Uematsu, Roman Untilov, Zdeňka Urešová, Lar- raitz Uria, Hans Uszkoreit, Andrius Utka, Elena Vagnoni, Sowmya Vajjala. Rob van der Goot, Martine Vanhove. Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Uliana Vedenina, Eric Villemonte de la Clergerie. Veronika Vincze, Natalia Vlasova, Aya Wakasa, Joel C. Wallenberg, Lars Wallin, Abigail Walsh, Jing Xian Wang. Jonathan North Washington, Maximilan Wendt, Paul Widmer, Shira Wigderson, Sri Hartati Wijono, Seyi Williams, Mats Wiren, Christian Wit- tern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan Arife Betül Yenice,
20

---

Published as a conference paper at ICLR 2023
Olcay Taner Yıldız, Zhuoran Yu, Arlisa Yuliawati, Zdeněk Zabokrtský, Shorouq Zahra, Amir Zeldes, He Zhou, Hanzhi Zhu, Anna Zhuravleva, and Rayan Ziane. Universal dependencies 2.10, 2022. URL /ttp://hdl...r http:/hhl..aanne... net/1 1234/1-4411 1-458 LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÜFAL), Faculty of Mathematics and Physics, Charles University.
Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzmán. How robust is neural machine translation language imbalance in mul- tilingual tokenizer training? arXiv preprint, 2022. httptpss//aaxii..rrr/bb/////////
14268
14268
Yukun Zhu, Ryan Kiros, Rich 7emel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015. URL https: /www.
htt.a//aaaii.ioooooooooocceeeoea papers//Zu Books and wtedd__cv___aa00 2015 aotoooss_aa___Cc_2211_pppppppp pdf
21

---

Published as a conference paper at ICLR 2023
A ABSTRACT RECONSTRUCTIONS
the of
numb
led
POIL
ther inder
Trom Br or
mods runde $388
topre
ropre
gapte
apric
lnd bat es ITE redeed of aa 180
SERE bon
of greniiing a destio bono esr en
rie ell 18 par net.
POXEL model
BUR anS
1K 590
BUR tong
mememin propppee esing
188
found
1gi &
found preeresnngda
P. with LLBS
than BERT Urtee the benel
BETT
moodhn: lan
than BERT. turtsr core the tor eftss moociing lan
Lnguugg i bode efite over larges of mout
moodl are wore Iiee set of ingun of
Language mocels are corico aa baggg inguns Wr
aaampote betller
allemp
laggages Be
lang
esented
computatoonl es
the emrbddiing mar and com.mmmutooon conoonaa cou out
ut layer, This pap H F pcuces PEIL 1et vetoase
paper PS the vol Encor
utt layer, paper Fn PEXEL, the Paol Dec- the
Lang
the
age
Lndl diiiig sett Tander represant.
manaderr gresentale
aanguag Da ontrry spriic
spric a Mamm or
58 ort og spripc
the cooot
preacting desrputionn
predicting
Sonove
ordtenge toutin
POXE
Enghi data BERT ande vol on
and eve ane
Englid dato
cnd uno sytaatic and momon
incuuing g roisre
passsinypp gomace guages, incuiing vor ourous
mn
onay emetiip tie light
emenikg scesonng
fou gheth
found in the tennegg
Heske than BEST then ling with br sor ets
oretwheen we with Lebn sceges
the BER when morinng with Labn sor d88 Fürther
more we tnd that nox mmor 1880 onn noun
UE fen that PExE nore benee to ro text nouts
more. Ind POKE momor to co text noutz
BERT tur commme g the mooding
o0hi ing 583 modeling lan
mmngggg the benk mooeling
puage
RE
atlemps
the number of somported angagees los repreb
The number pages lacking the aul te
the rn bbergl supported lang paggesss. locking Uhi
the
skne
tweene rep led
resull the between
thonal cou
sufferst Bostoas
POXEL
of Lang age, whtch suffers Tomrarther of
of whtch frome B profthe::
& calned lang model that randers
andd age whathha
38 modi run Bers
guagesta possibl
9100 tog pepepresa
ges
represa
any
hed patch es of preacteen and dust
intead produeng detrio Bon
patc irtee preaiting ddtritution
en
We pretr en once
par
the
per are get PEXE mode bed Ute
gugggs
Eggasy Pice salnien dy
Laimoenpe. We musd
o uuRtbeedd badi memeii essing ast lound
BUR on yeaate and semenik ann ash
(and data 50 Further
found the gret tatiggaaa PIKULE
teakerthen PPKK,
reakerth
working
PDX tor 38538 oockig lan
POXE
inan BI BT turthrr conni manginn the benettt to modeling &
than BE fur
than HRT turth er co tomiingth cenet eo modeing bn
Language mocdls are rio ovore tine of mouns
- moodss are dod oo set maurg off
Languoge mooclo of deante OF a inuuns of
emp
allempi
botlee of bet
supporte sagee,
off gesa
egresented
embee dang matro and
our
the matea and
our
the errbedding matro and computatonal oou thehe out
ut byee, This paper proouu as PEKa. the gell
This ard uces Fox based Enoor
ut byyer, This paper Int prrouuues PEXEL, the
Language suffee
ther as
Lang
braennege
miigng poss by iaander opresantation
posste fandeer reprosatation
anguages hased ont spri ppresantations
making
gugggss la an ont tn appppiic Monyys co
booe ontron spric amannyy or th
the co
of paods
rnrersttaa the BB
thes inte in
presicing
Bono
BBM
FOXEL
We
Enga: doto BF0 and eve syntame and
dam and oud ane 96 on an
and
moconmaooo call dhorse ong orgunges, rruiing varouse krsensss Wel
onguagos, nousing vor ouross
hat antaee found emmnktelss
onayitani
nnttonngee df the
inng
6100 ligh
data PIXLL
weaker 1aa BORTwher 98 ling with soege
than BERT then with Le 501
whee wertempiin Labn ser pts Furthe
more tnd that PXXE. hore smmento rosr text nouth dt
man that PD momeereo r00 roor text nott
more fnd that POXE more
then BERT
the cene t5 moodiing
comninigg benett: of moodinng lan
th
ring the ponett mooeling lan
Figure 4: PIXEL image reconstructions of the abstract with different span masks.
22

---

Published as a conference paper at ICLR 2023
RECONSTRUCTIONS RECONSTRUCTIONS
B WEB TEXT
message si mpleet be cause truly Our message is si mplar
we truly bas
Our message simple: Be cause truly
amntto our peanut-lovin hearts that peanut
in our peanut-loving hearts that peanut
ve with our peanut-lovin loving hearts that peanut
make everyth iing sadd Peanuts are perf
make everything betted, Peanuts are perf
make everyth ing better, Peanuts are perfe
ctly peckeadd because they're packed with r ctly pommrtid because they're packed with
ctly powerfül because they're packed with r
osation and they bring people togethe: Ou o vision and they bring people togethe Ou
ffction and they bring people togethe: Our
thirst
knowle dge unquenchab thirst for peanut know dow unquenchat
thirst orr peanut know dge is unquenchab
ee Werre always sharing snackable news st
50 werre always sharing snackable news
ee 50 were re always sharing snackable news st
pries
benefits peanuts.
pries about the benefits of peanuts eeaannts, nemmu pries about the benefits peanuts, its
ttats research, etc. our passiondfor peanuts tats, research, etc. Our passion pe peanuts tats, research chch etc. our passion for peanuts
infectious. We bemrane peanuts as if they v s infectious. We looks at peanuts
they v
infectious, We broke peanuts as if they
ere home (un away from wir ning ttmeswe ere a home (un away from wir ning Welon't
ere home (un away from wir autiie don't
care about peanuts and the people who geve care about peanuts and the people who goo care about peanuts and the people who go
the m. We give shout-outs to those who a the
give shout-outh
who lift
the m. We give shout outs to those who
up and promote reainutsinn and the peanut sta
up and promote pe inuts and the peanut str
up and promote pe inutt and the peanut str
rry. We're an horitrity on pear gts we'r rr.. We're an horitrtty pe aaautt and
pry. We're an authority on pear uts we'eer
anything but boring.
anything but boring.
anything but boring.
Penguins are designe to be streamined an Penguins are designe designed to be stre streamineed an
Penguins are des designed to be streaminedd an
hydrody wodyyiunkkkkkkk hav ing tieeting gs would
d hydrody namk, so having long legs would
hydrody nami, so having long legs would
dd speading Hav ing short legs with w wenae
dd e expamingg Having short with wende
dd expanding Hav ing short legs with w wodde
feet to act like runders, helps to give them
d feet to act like runbers, helps to give them
d feet act lkke rubbers. helps to them
that té do like figur e Momnntoompar
bire
that thededo-liie figur tsnseo compare bird
that thrmedo doneedolio loklike figur t. we compare bird
anatomy with humans, we would see s omet
anat omy with humans, we would see omet
anatomy with humans, we would see s somet
ningins speculiar, By taking look
side
ning mog peculiar. By taking a look the side
ingg aist paculiar. By taking look
side
by- iing image in Figure e 1. ou can see how
by- in hage in Figure e 1, Du can see how
by iing image in Figure 1. yu can see how
their leg bones oramege to ours. What most
their leg bones ore mage
ours. What most
their leg bones aempos to ours. What most
people mistake for knne are actually the and
people mistake for knee es are actually the an
people mistake for knees are actually the ar
aato bi ds. This ves acccclssion that b
eommes of bi ds This g ves the clusion that b
atommes of bi ds. This ves the illusion that b
ard knees
opposite ours. knees
knees bend opposite ours
knees
idd knees
opposite ours.
knees
aee adually tucked up insidl the bokes
re a duullyy tucked up insidle the moxesmonn aee adtually tucked up insicl the bousen
the how does this look ins de the
of the bird So how does this look ins de of a
of the bird So how does this look ins de
benguin? In the bendle below, you cai see b
henguin? In the arages below, you can see b
bengun? In the images below, you can see b
oxes surrounding the penguins knees
pxes surrounding the penguins keees.
oxes surrounding the penguins knees.
righ the ir fig
maly lidral therger th eie Prylis
ahus anst puoi at de aaaadd Chemma For me
inha cescaane pook
woak-hhamm For ahener th
rchs ce anst poak at de haaade chemma For cordenter ih
mestt notchrough a aeorrl anggage but in 1
y communcate notchrough speefi ang but (36
communsstte notchrouch augha a spedic inggage but Io eledrial Booped
redriaal pooped
specil
eecricaal
specil genektrrs
ungreddartoo
The
with sadi) divicaal copali amnmming one
with bad diviiaal copall arouunng) ot eketr
with eadi) divicaal
arouding one
B surguu our aotonnus sg v.
oioe surggu out aotoonuus sagggg a. IIS expn aeor
okee sungqu monnnoss sg oven ampor codee"
Deme Cather ledlogad
hellogs Warh angen dieoom us
nieesll Luaay
nieessle Uuaay
best one
thour
enon tre
sponka inte se ie
Anon /ni
speaka Inte 1r le BaaadO Obam
enonyynu bo Renememeenn sponka into le Barack Obam We cerenthoyume uant send an inportant message. They sto amament gago de
eeennhyyuune uont to serd on porrrtnt message. They st
eeenthey vant to send an port
Trey st
ammmeet. those gapo
part
ja aimoment those gapd d very part
Tig "ighnforeatin restonmmmen aboud g operity coursse
readon abou whmmer previtent,
counse
The reaiinnnnnn monoe about g geeperty or courtst iin heve, thn tomeer previlent.
Gerhen sed Farmi inss here, (hn Tommer prevdlent
snd Facene
mester of the at afeee: dro 188 pause- Amoomend trck th
mester od the on afeee d 188 pause Auacmeag trik
mester d the on dfee d 188 pause aa0cnaas tnck dh
can
strongy to 6089
lp
strongy negeers epo til
strongy hellle pubkited moda Commert Bangy
can ee
moda in conmart 8 Bongg
publishe moda in Cumfont
(a) 100k steps
(b) 500k steps
(c) IM steps
Figure 5: PIXEL image reconstructions after 100k, 500k, and IM steps of pretraining. We overlay the masked original image with the model's predictions. Images are wrapped into squares and resized for visualization purposes only. The texts were not part of the training data. We see that the fully trained PIXEL (IM) predicts masked spans more clearly and accurately. For longer spans with a larger possible prediction space, multiple predictions may appear together creating blurred text.
Reconstructions of three sources of tex figure also shows how PIXEL (visually) expresses uncertainty, e.g. for reconstructions of long spans where the space of possible outputs is much larger than for short spans, and how it captures long- range dependencies. In the third row, we can for instance see that PIXEL uses context from the beginning of sequence (Barack Obama) to correctly fill in a gap later in the sequence, and vice- versa (Brienomyrus).
after 100K, 500K and 1M pretraining steps. The
https: hwtp://wwwwpppoooeeaaaaa///////e frequent (reuuuntty-sskkddquuuttttii////r
http::/www.eelliiiee.eeeeeeeeete brttps://www.thhe
23

---

Published as a conference paper at ICLR 2023
C CODE
PIXEL is implemented in PyTorch PPazkke al., 2019) and built HuggingFace transformers (Wolf et al. 2020). We make our code available at nttpp::/githhb.. com/ httpm//ppppppelllllppppppllllple Our pretrained PIXEL model, including large number of intermediate checkpoints, is available at htapss:///gggnngfaacccc www.e..ooeeeeeeoooeeeeeeeeeeoooe cwt.eem-PPXEEL/iiiaiicee-oo tiple seeds each, are available through the model hub.
and our finetuned models, including mul-
D TEXT RENDERER DETAILS
Rendering backend We experimented with different text rendering backends. Following Salesky et al. (2021), our first implementation was based on PyGame which PIXEL was also pretrained with. Later on, switched backend based Pango TTylor 2004) and Cairographics which has native support for complex text layouts, making it possible to specify fallback fonts, and has faster rendering speed. Without fallback fonts, we would be limited
maximum number of 2 - glyphs that can fit into single OpenTTppe or TrueTTppe font file due to a technical limitation 24 By leveraging fallback fonts, can theoretically cover all Unicode codepoints, including emojis.
Fonts We rely on the Google Noto Sans fonts collection which covers the majority of Unicode codepoints and is actively growing 26 Note, however, that PIXEL is compatible with any font and can therefore encode anything that can be typeset on a computer screen. We used font size of 8 at 120 DPI for pretraining with PyGame, which rendered height of 16px. It can, however, also be adjusted at finetuning time. For finetuning with PangoCairo,
selected manually fit most scripts into
a font size of 8 (120/72) As 13.33 which yields roughly the same outputs as the PyGame renderer. Due to how glyphs are shaped by the two backends, the outputs of the two renderers do not exactly match. Because did not employ data augmentation to make PIXEL robust to such changes in font size, we recommend using the PyGame renderer it was pretrained with for zero-shot applications with PIXEL. When finetuning, this minor mismatch in rendering outputs easily overcome by PIXEL, generally recommend using the PangoCairo renderer.
Characters glyphs For extractive QA, it is necessary to obtain mapping between the characters in the context paragraph and where they appear on the rendered image. Obtaining this mapping is straightforward due to how text is rendered. The shaping step the rendering pipeline converts characters into glyphs[F7 27 In ligatures, as common for instance in Arabic, glyph composed of multiple characters. Likewise, an emoji often consists of a base codepoint and modifier codepoint (c.g. to change the emoji skin colour) which are represented by a single glyph. For accents, on the other hand, one character might yield multiple glyphs 28 In practice, the renderer therefore uses grapheme clusters, whose logical boundaries in the rendered image we can map to the input characters For simplicity, we assign each codepoint a grapheme cluster to the logical horizontal offset at which the cluster starts on the rendered image. Future work may investigate alternative mapping strategies.
RGB rendering PIXEL supports RGB rendering which may be useful to accurately represent colour emoji and for multimodal applications in the future. However, 24-bit RGB rendering is slightly slower than 8-bit grayscale rendering (see Table below) for text written in Latin script, which is why we made RGB rendering an optional setting. In our pretraining and finetuning experi- ments we rendered text in grayscale, and we generally recommend doing so when not working with coloured inputs.
org/
org/
24 See
for an explanation.
i/ptps::/fonts.gogglee
26
http:///atttfoott.....bobboooooe an overview of Noto's Unicode coverage.
for an overview the rendering pipeline.
hhttps://doss.gte
httpss///macdddeer////rrrrtttttt Cluuster Boundaries
Cl
24

---

Published as a conference paper at ICLR 2023
Right-to-left scripts PIXEL's renderer natively supports right-to-left (RTL) writing. In the default setting, the base text direction (which for instance determines on which side of a sentence punctua- tion marks are placed) is inferred automatically by the rendering backend based on the first "strong directional" character
given paragraph 30 The mirroring of RTL characters is also handled au- tomatically according to their Unicode bidi attributes. Optionally, the base text direction can be set manually, which is useful when working on monolingual data, c.g. in Arabic or Hebrew, as the renderer does not have to go through the direction check. In S we describe limitations of how currently handle RTL writing.
Arabic
Chinese
English
0.03
0.04
0.04
Proportion
0.02
0.02
0.02
0.01
Finnish
Indonesian
Japanese
0.08
0.04
0.08
Peoportin
0.05
0.05
0.02
0.03
0.00
0.00
0.00
Korean
Russian
Turkish
0.06
0.04
0.30
Reference
Proporion
BERT
0.04
mBERT
0.02
PIXEL
0.02
50
1.50
50
50
100
Longth [Tokens/Patchhe]]
Length [TTkknn/ Patches
Longth [Tokon</atchee)
Figure 6: Distributions of sentence lengths from monolingual UD corpora after tokenizing by BERT and MBER and rendering by PIXEL, compared to the reference by UD treebank annotators.
Processor
Batched
Throughput (ex s]
ENG
ZH10
Renderer (Grayscale) Renderer (RGB)
X
3944.1
6309.0
x
3615.1
6849.5
19128.9
18550.5
Tokenizer (Rust)
4782.9
5684.4
1286.6
2637.1
Tokenizer (Python)
X
1286.8
2580.9
Table 6: Throughput comparison between PIXEL's 's PangoCairo renderer and the fast and slow BERT tokenizers, implemented in Rust and Python respectively, from the HuggingFace tokenizers library. We estimate throughput, measured in examples per second, by how long it takes to process 1M lines of English (ENG) and Chinese (2110) Wikipedia text on the same desktop workstation (AMD Ryzen 9 3900X 12-core CPU). We distinguish between tokenizing all lines individually (Batched X) and as one single batch (V).
Efficiency analysis We briefly analyze the text processing (rendering versus tokenizing) efficiency in terms of a) length of the processed sequence, which has direct effect on GPU memory consump- tion and the time it takes to compute forward and backward passes, and b) processing throughput.
For a), follow Rust et al. (2021) and process the training and validation splits of all available UD v2.10 treebanks in various languages with the PIXEL renderer and the tokenizers BERT and MBERT. We plot the resulting sentence length distributions in Figure including comparison
See http:://uucccoe..o////rrtsssssss for overview of the Unicode hidi algorithm.
25

---

Published as a conference paper at ICLR 2023
with the reference segmentations from the UD annotators. For English text, the PIXEL renderer is slightly less efficient, i..., it produces slightly longer sequences on average than the tokenizers. For other languages with Latin script, e.g. Finnish and Turkish, the renderer is more efficient than the BERT tokenizer, alheit slightly less efficient than the MBERT tokenizer. For non-Latin scripts such Arabic and Japanese, we see that the renderer
be lot more efficient than both tokenizers. The English BERT tokenizer is technically fairly space-efficient for non-Latin scripts but this is misleading because it largely produces [UNK]s (recall right side of Table T and each [UNK] a single token; the functionality of the BERT model on a sequence of [UNK] is strongly compromised.
For b), we compare the processing throughput of HuggingFace's ' BERT tokenizers and our PIXEL renderer in 6 We find that the Rust-based BERT tokenizer with batch processing achieves the highest throughput by leveraging parallelization. When not using batch processing, it comparable in throughput with PIXEI'' 's renderer, i.e. depending on the language or script, rendering can be slightly slower (ENG) or faster (ZHO) than tokenizing. Since the rendering backend (PangoCairo) is implemented in C, we expect to achieve similar gains in rendering throughput by also leveraging parallelization for batch processing (in contrast to the Python-based tokenizer which is limited by Python's 's global interpreter lock (GIL)). We plan to implement batch rendering functionality in the future.
E ARCHITECTURE & PRETRAINING DETAILS
PARAMETER
VALUE
Image size
(16, 8464, 3)
Patch size P
Encoder hidden size Des
768
Encoder intermediate size
3072
Encoder attention heads
Encoder num layers L.
12
Decoder hidden size Dax
512
Decoder intermedhute
2048
Docoder num attention heads Decoder num layers, Layer norm (Ba et al. 2016
16
le-12
Span masking ratio π
0.25
Span masking max length
6
Span masking cumulative weights W
00.2,0.4,0.6.000 0.8.0.
Span masking spacing
Dynamic
Dropout probability
0.1
Hidden activation
GeLUU (Hendrycks & Gimpel 2016
Optimizer Adam
Adam [Lostcintov & Hulter 2019 Kingma & Ba 2015
(0.9. 0.99).
Adam
le-8
Weight decay
0.05
Peak learning rate
1.5e-4
Learning rate schedule
Cosine Decay (oohchilov & Hutter 2017
Minimum learning rate
le-5
Learning rate warmup ratio
0.05
Training steps
IM
Batch
256
Table 7: PIXEL pretraining settings
Patch Embeddings PIXEL reshapes each image æ into a sequence of = W/P non-overlapping flattened 2D patches y E Nx(P(C)) where / = 16 is the patch size, and linearly projects them via E E R(P Dax obtain patch embeddings 2r, E.) E Nx with encoder hidden size Denc P2C 768 3T Afterwards, fixed sinusoidal position embeddings Epus E (N+ Dew
are added, leaving out the position vector in position 0 for a classification (CLS) embedding later: o x, + [Epos.... Epss ((N+1)) +1)
31 This is equivalent to projecting each rendered image æ E ig XX via a 2D- cocoovvllli layer with C' input channels and Den: output channels and kernel size and stride hoth equal to the patch size P, which we do in practice.
26

---

Published as a conference paper at ICLR 2023
1.0
0.8
Loss
0.6
0.4
0.0
Training steps [K]
Figure 7: PIXEL pretraining loss curve
Span Masking PIXEL then masks out R = 25% of the N 529 embedded patches via span masking with max span length S = 6 and cumulative span weights W = {0.2, 0.4, 0.6, 0.8, 0.9, 1}. ie. B(s)) 3.1, as outlined in Algorithm Applying the mask M, we obtain the unmasked patches
Vis [ M)
Encoder Following ViT-MAE (He et al. 2022), the PIXEL encoder only operates on unmasked patches (i.c., 396 patches at 25% masking) and a special CLS embedding with its positional encod- ing [(as)] Ep R is prepended to the sequence: hg [c. vvis
R [R(+LRR-NJ)) Pas 32
Let (h, be the encoder hidden states after each of the T. = 12 encoder transformer layers, and
hg denotes the input sequence. The outputs of each transformer layer computed detailed in
Vaswani et al. 2017 33 and the last layer's output 12 E [R[++RRN]] Des is passed to the decoder.
Decoder The PIXEL decoder first projects the encoder outputs via Edec E Pa x Dao to obtain decoder embeddings za h2 Exec (R-N))) Das, where Daec = 512. Next, mask embeddings [[mask] E 1 Dax are inserted at the masked-out positions and fixed sinusoidal position embeddings M) + Epos] E [[(++)xD&x { are the decoder hidden states after each of the K = 8 decoder transformer layers, computed in the same way as the encoder hidden states, and dg denotes the input sequence. There is encoder- decoder cross-attention. The decoder output dæ E (g)++))xDa projected via E P X(FC) to obtain patch-wise logits o = (dxO) E I (N++1))(P9)) Finally, the CLS logits are removed and a normalized mean squared error (MSE) pixel reconstruction loss is computed: Cnompix QQ Eieq |normalize (x=) o¹12 with denoting the indices in the set of masked. non-blank (text) patches Q {i (M 7) and normalize(-) dividing the difference between the target patch and its mean by its standard deviation.
are added obtain do [(x U [minss]]]
F FINETUNING DETAILS
Table 8 gives an overview of all languages used our finetuning experiments, Table glinks to our finetuning datasets, and Table T0 lists the UD treebanks used.
We list our finetuning recipes in Table II for POS tagging, dependency parsing, NER, QA, and XNLI and in Table 12 for the GLUE tasks. Due to compute limitations did not comprehensive hyperparameter sweeps. Instead, we relied on sensible priors from finetuning BERT and made slight modifications needed. In most cases, hyperparameters that work well for BERT also work well for PIXEL. For some of the semantic tasks, in particular NL.I and SST-2, we found that some random initializations did not converge. In those cases, minor tweaks the learning rate or increasing the batch size usually helped. For GLUE, we found that PIXEL performed slightly better on some tasks with the PangoCairo renderer, whereas for others, using the PyGame renderer (which PIXEL was
In pretraining, no loss is computed for the CL.S embedding but it can optionally be used when finetuning PIXEL for sequence-level downstream tasks.
Note that encoder and decoder do not attend the blank (padding) patches that appear after the EOS patch.
27

---

Published as a conference paper at ICLR 2023
pretrained with) was more stable. We plan further optimize the training recipes and study PIXEL's convergence behaviour in the future.
For word-level tasks, add padding in order to render each word at the start of patch and so create a bijective mapping between words and patches. Doing so assumes that word boundaries available. We note that subword-based and character- based models also make this assumption. In BERT, for instance, word-level tasks are formulated such that a word's label as- signed to its first subword token, requiring word boundaries. During training, continuation tokens are then masked out when computing the loss. Consequently, predictions for continuation tokens also need be masked out inference time, which again requires word boundaries or aggregation strategies that may introduce errors. The same applies to character- based models. For PIXEL, should still possible to render the text without adding spacing, although the
image
this assumption be violated, mapping is then no longer bijective as multiple words can overlap on one image patch. In such cases, assigning the prediction for patch to either word can cause loss of information. Although prac- tice this approach does not necessarily affect performance negatively, future work will investigate alternative approaches.
Language
ISO 639-3
Language Family
Script
Amharic
AMH
Afro-Asiatic
Ge'ez
Arabic
ARA
Afro-Asiatic
Arabic
Bengali
BEN
Indo-European
Bengali
Bulgarian
BUL.
Indo-Furopean
Cyrillic
Chinese
ZHO
Sino-Tihetan
Chinese
Coptic
COP
Afro-Asiatic
Coptic
English
ENG
ando-uurppean
Latin
Finnish
FIN
Uralic
Latin
French
FRA
Indo-European
Latin
German
Indo-Europe auro-european
Latin
Greek
ELI
Indo-Furopean
Greek
Hausa
HAC
Afro-Asiatic
Latin
Hindi
HIN
Indo-Europe
ando-uuoppean
Devanagari
Igbo
IBO
Niger-Conge
Latin
Indonesian
IND
Austronesian
Latin
Japanese
JPN
Japonic
Japanese
Kinyarwanda
KIN
Niger-Congo
Latin
Korean
KOR
Koreanic
Korean
Luganda
Niger-Congo
Latin
Luo
LUO
Nilo-Saharan
Latin
Naija Pidgin
PCM
English Creole
Latin
Russian
RUS
Indo-Europeat
Cyrillic
Spanish
SPA
Indo- European
Latin
Swahili
SWA
Niger-Congo
Latin
Tamil
TAM
Dravidian
Tamil
Telugu
TEL.
Dravidian
Telugu
Thai
THA
Kra-Dai
Thai
Turkish
TUR
Turkic
Latin
Urdu
URD
Indo-European
Perso-Arabic
Victnamese
VIE
Austro-Asiatic
Latin
Wolof
WOL
Niger-Congo
Latin
Yorúbá
YOR
Niger-Congo
Latin
Table 8: Overview of languages used in our experiments.
28

---

Published as a conference paper at ICLR 2023
Dotaee
Download Link
Reference
Dependencies 2.10 MasakbaNER
htips://
cz/reposi http:/looy/////aaeeeeeeeeeee
Retos:/ / /Tgt cono nasakranne 137 akhann ner www.//rrrr///ee data
Poceia e
2041
GLUE
Brtal: 7 /rriggingt ace car natasso E5/
Wals mr
2015
TyDiQA-GolIP
SQUADvI.1
Brtps: /t/as///mgggggg are oalase squae
Kajurkka 2016
KerQuAD 1.0
Io 7rutern aue c9roal Lase 13 Sined kor
Lin ai 2019
JaQuAD
XNIJI
https://nigging natasettteee. X011
Conteau
2018
Table 9: Links and references to the datasets used in our finetuning experiments.
Language
Treebank
#Sentences
Reference
ENG
English-EWT
16621
Silveira
2014
ARA
Arabic-PADT
7664
Hape et al
2009
COP
Coptic-Scriptorium
2011
Zeldes Abrams
HIN
Hindi HDTB
16647
Paliner cl al 400
JPN
Jepanese-GSD
8100
Asaham et
KOR
Korean-GSD
6339
Chun al 2018
TAM
Tamil TTB
600
Ramasamy & Zabokrtsky 2012
VIB
Vietnamese- VTB
3000
Nguyen et 2009
ZI1
Chinese-GSD
4997
Shen ét al. Y010
Table 10: Overview of the Universal Dependencies v2.10 (Zeman et al. 2022 Nivre et al. 2020) treebanks used in our POS tagging and dependency parsing experiments with the number of sen- tences in their respective training splits. As mentioned in $3. these treebanks were chosen with typological and script diversity in mind.
PARAMETER
POS
DP
NER
QA
XNLI
Rendering backend
PangoCairo
Classification head pooling
Optinizer
AdamW (0.9. 0,999)
Adam
Weight decay
0
Leaming
So
[Sc
Se
Se
[3c
Sc
5]
2c 5
Leaming rate warmup steps
100
100
100
100
Leaming rate schodule
Lincar docay
Max sequence length
256
256
196
400
196
160
Batch size
64
64
64
32
256
Max steps
15000
15000
15000
20000
50000
Farly stopping
Eval stops
500
500
500
500
1000
Dropout probability
0.1
Finetuning settings for POS tagging, dependency parsing (DP), NER, QA, and XNLI. We did not run a comprehensive hyperparameter search due compute limitations; these settings were manually selected based on small number of preliminary runs. Maximum performance was often reached well before the specified number of max steps.
Table 11:
PARAMETCR
MNLI
QQP
QNLI
S8T-2
COLA
STS-D
MRPC
RTB
WNLI
Readering backend
PangoCairo
Py Game
PangoCairo
PyGame
PyGame Mcan Adam
PyGame
PyGame
Py Pame
PyCame
Classification bead pooling
Optimizer
Adam P
(0.9, 0.999)
Adam
Weight decay
Leamning rale
3c-5
3e--5 100
3e-5
3c-5
2e-5 5
2e-5
3c-5 100
3e-5 200
1c-5
Learning rate warmup stops
100
100
100
200
100
100
Learning rate schedule
Lincar decay
sequence length
Batch size steps
64
256
64
256
256
64
64
64
256
15000
15000
15000
15000
15000
Early stopping
interval
steps
steps
500 steps
stops
steps
steps
steps
steps
epoch
Dropout probability
0.1
Table 12: Finetuning settings for GLUE tasks. We did not run a comprehensive hyperparameter search due to compute limitations; these settings were manually selected based small number of preliminary runs. Increasing the batch size to 256 and switching to the PyGame renderer helped achieve more consistent convergence behaviour for some tasks. For the smaller datasets (to the right of QQP), maximum performance was reached well before the specified number of max steps.
29

---

Published as a conference paper at ICLR 2023
G EXAMPLES OF Zeroé ORTHOGRAPHIC ATTACKS
Sentence
NoNE
Pergains are designed to be streamlined
CONFUSABLE
Penganns are dessggned
streswlined
SHUFFLE (INSER)
Pegnuins are dnesiged to be sieatrmaled
SHUFFLE (FULL)
ngePaaios rw ulgaalmci to be etimaslernd
DISEMVOWEL
Pagns r dugnd to be strmind
INTRUDE
lenguiss afre dessinnee
strecamlined
KEYBOARD TYPO
Pergainz are dwsigned ro ne strcamilinal
NATURAL NOISE
Penguijs ard design ti bd strcamlinfd
TRUNCATE
Penguin designe to streamline
SEGMENTATION
Penguinsaredssign wwwpreddeiigooooeeeennnnnng Pengwains' 's ar chiseind to storimlignd
PHONETIC
Table 13: Examples of low-level orthographic attacks based on the Zeroé benchmark.
Caeihable
Disrmowel
Shuffe (futh
Stuffi (brer))
Irrade
60
Accing [N)
70
80
50
Kerboad
Aameog [N]
80
70
Modal
BERT
40
50
80
Figure 8: Test set accuracy for a single run of PIXEL and BERT across different levels of noise introduced through various orthographic attacks in SNLI. The results show that PIXEL is more robust than BERT to most of these attacks.
Confusable
Disemnowel
Shuffle (full)
100
Accuracy [36]
75
50
Shuffle (inner)
Intrude
Keyboard typo
100
Accuracy [16]
75
50
Natural
Phonetic
Truncate
100
Accuracy [9%]
75
Model
BERT
50
PIXEL
20
50
0
80
20
50
80
Figure 9: Test set accuracy for a single run of PIXEL and BERT across different levels of noise introduced through various orthographic attacks in POS tagging. The results show that PIXEL is more robust than BERT to most of these attacks, especially when dealing with cisually character substitutions. SEGMENTATION is not applied to the task of POS tagging, since the joined words would not have proper tag.
30

---

Published as a conference paper at ICLR 2023
H FONT TRANSFER ANALYSIS
In this section, we analyse the adaptation capabilities of PIXEL to new fonts finetuning time. Specifically, we finetune PIXEL models for POS tagging and dependency parsing the UD UD_Englinnn EnglishhEWT treebank and sentiment analysis on SST-2, once with a font similar to our GoNotoCurrent / PrtoSans-Regulae pretraining font, NotoSerif ioerif-ffo Regeularr and once with font strik- ingly different from it, JournalDingbatsl. We compare the three fonts in Table 14 below:
Font
Rendered Example Sentence
GoNotoCurrent
cat loves oatmeal and pancakes. cat loves oatmeal and pancakes.
NotoSerif Regulaa
Journal Dinghats1
Thule 5 0:ire arr Twett T
Table 14: An example sentence rendered in three different fonts.
GoNotoCurrent
NotoSerif-Rest f-Regular
JournalDingbatst
POS
96.7
95.9
93.9
DP
90.6
88.1
81.3
SST-2
89.6
84.2
72.9
Table 15: Results for fine-tuning PIXEL for POS tagging. dependency parsing (DP), and sentiment analysis on SST-2 with three different fonts: the font used in pretraining (GaNotoCurrent), visually similar font (NotoSerif Senteeif-Reegee -eRgular), and a highly dissimilar font (JournalDingbatsl) We report test accuracy for POS, test LAS for DP, and validation accuracy for SST-2, each averaged over runs.
The font transfer results are shown in Table 15 ability out-of-the-box. i.e. without any font or image augmentation strategies employed during pre- training In line with expectations, transfer easier than to dissimilar font (JournalDingbatst) Nevertheless, PIXEL is able to transfer surpris- ingly well to the JournalDingbats! font, in which every letter is simply mapped to the icon of an object or animal.
We find that PIXEL exhibits fairly high font transfer
visually similar font (NotoSerif-Ren if-Regular) is
I FURTHER ANALYSIS
To investigate where PIXEL currently lags behind BERT, we analyse the impact that dependency length has both models in dependency parsing in ENG. We can see in Figure 10 that the LAS gap between BERT and PIXEL increases with longer dependencies, indicating that PIXEL struggles slightly more with long syntactic dependencies.
#1
ENG
ARA
BUL
DEU
ELL
FRA
HIN
RUS
SPA
SWA
THA
TUR
URD
VIB
ZHO
MOEETT
179M
83.3
73.2
77.9
78.1
75.8
78.5
70.1
76.5
79.7
67.2
67.7
73.3
66.1
77.2
77.7
110M
83.7
6488
69.1
70.4
67.7
72.4
59.2
66.4
72.4
62.2
35.7
66.3
54.5
67.6
46.2
PIXEL
86M
77.2
58.9
66.5
68.0
64.9
69.4
57.8
63.4
70.3
60.8
50.2
64.0
54.1
64.8
$2.0
Table 16: Results for PIXEL and BERT finetuned on XNLI the comnslate-train-ee setting where train the joint training data in all 15 languages, originally translated from ENG by Conneau et al. (2018). We report test set accuracy averaged over 5 runs each. Despite the relatively large performance gap in favor of BERT in ENG (which is in line with the GLUE results in Table 33) the gap is much smaller for other languages, particularly those not using the Latin writing system. PIXEL overall more consistent across scripts, outperforming BERT THA and ZHO.
'e believe such augmentation strategies would further improve robustness to font variations and leave this experiment future work. Considering that we have full control over the font when working with NI.P text datascts, robustness font variations primary goal in this work.
31

---

Published as a conference paper at ICLR 2023
Laboled Attochmment Soore (1%
95
90
85
Model
BERTaASE
75
PIXEL
70
Distance to Head [ words]
Figure 10: LAS scores (ENG) across different dependency lengths averaged over random intitial- izations of BERT and PIXEI.. In ENG, long syntactic dependencies are more challenging for PIXEI..
J LIMITATIONS
This paper introduces a new approach to processing written language as images, which removes the need for a finite vocabulary, providing solution the vocabulary bottleneck. While our results show that PIXEL is a promising approach in this direction, this is only the first step. Here, we highlight current limitations and avenues for future work for pixel-based models:
PIXEL is pretrained on predominantly English text written in the Latin script. The choice of English is driven by the scientific goal comparing against a widely used model (English BERT) but English may not be the best source language for cross-lingual transfer (Ture et al. 2021 Blevins et al. (022). We expect that PIXEL trained on typologically diverse languages in multiple scripts would considerably surpass the cross-script and cross-lingual transferability of English-oni PIXEL but this remains to be verified, and training model on large amounts of data will require large computational resources.
PIXEL currently seems to be less sample-efficient than subword-bard based PLMs. PIXEL ex- cels at syntactic tasks after being pretrained for the same number of steps/datapoints as BERT (a challenging setup within an academic budget), but still lags behind semantic processing, As a consequence, it also requires more training steps than BERT to converge during finetuning. Closing this gap might involve longer pretraining with additional (long- dependency) objectives.
There challenges be addressed when working with languages written right-to-lect leftt PIXEL currently processes sentences in such languages from the end to the beginning which may lead to learning inadequate features for sentence separation and position embeddings.
PIXEL cannot be used for language generation tasks because it is not possible produce discrete words from the pretrained decoder.
Rendering text as images requires more disk space than reading text from file. This can be alleviated by caching the dataset compressed format, or rendering the images on on-the- fly. Rendering images on-the-fly will create additional overhead when training for multiple epochs.
32


Reviews:
[{"review_id": "tdLv6WPXJc", "reviewer": "Reviewer_HZPD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors developed a clever twist to the traditional neural language modeling paradigm by building a model that operates on image patches of the text. It learns to output “masked” patches using an MSE loss. Through this, the model alleviates issues with fixed or non-transferable vocabulary. When fine-tuned on a suite of downstream syntactic and semantic tasks, PIXEL observes commendable performance often on par with BERT etc. or better for languages whose writing scripts are unfamiliar to BERT. The authors also show that the model can handle code-switching and works well in the face of character-level noise.", "summary_of_the_review": "Given the issues with vocabulary and tokenization in NLP, I think this paper offers a very innovative solution and I am curious to see how this research direction pans out.\n\nNB:I would give this paper a 9/10 if the option existed.", "strength_and_weaknesses": "Strength:\nI commend the authors for the innovation here. Although a simple idea, it is highly effective and clever, and presents a very exciting alternative to BERT & friends. I look forward to future research on what these models encode and share across languages/writing systems.\n\nQuestions:\n- It would be useful to have a more detailed discussion on the fertility of the pixel maps, ie., how many patches correspond to a word on average and the relationship to some of the tasks. For example, I am curious if there is a relationship between this and the question answering performance of PIXEL as the authors allude to as well.\n- It would help to order the languages in the results tables based on [UNK]%. and insert a row showing the difference between BERT and pixel for each result table.\n- Any hypotheses for why its performs much worse than BERT on COLA specifically?\n- I understand the space and time constraints but I would be excited to see that perhaps training PIXEL on two very different script systems leads to big performance gains than traditional models and PIXEL trained on english only. Especially if the other language has a rich morphology for example.\n- How hard was hyper-parameter tuning for this model? what do the loss curves look like?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "skNE86so_OZ", "reviewer": "Reviewer_znCA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors proposed a new methodology by rendering text as images, making it possible to transfer representations across languages based on orthographic similarity or the co-activation of pixels. The pre-trained step is learning reconstructing the pixels of masked patches instead of predicting a distribution over tokens.\n\nIn the empirical study, authors find that PIXEL substantially outperforms BERT on syntactic and semantic processing tasks on scripts that are not found in the pretraining data, but PIXEL is slightly weaker than BERT when working with Latin scripts.", "summary_of_the_review": "Overall I feel this paper is novel and has strong real-world application. However, my minor concern is the novelty is limited given visual text representations is already proposed by Salesky et al. (2021). Therefore I would recommend \"marginally above threshold\".", "strength_and_weaknesses": "Strength\n\nIt is interesting to see authors propose to render the text to image for language modeling. \n\nI believe the robust to orthographic attacks and code-switching will be very useful in web security domains. For example, such attack pattern very common in phishing email as the fraudster try to bypass the classifier.\n\nWeaknesses\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iAGgbBUGPKt", "reviewer": "Reviewer_zqCz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Pixel, a masked language modeling approach similar to BERT that first renders text as an image and then encodes the sequence of patches using a ViT. During pre-training, the setup resembles the masked autoencoder – spans of patches are masked, only unmasked patches are encoded and a decoder is used to predict the masked patches. During fine-tuning, the decoder is replaced with a classification head similar to BERT. Pixel is motivated by the need to remove the need for large vocabularies that increase model capacity via embedding layers and make prediction layers very expensive, while being able to generalize to a large set of scripts and languages.\n\nExperiments include pre-training Pixel on the same data as BERT (Wiki + Books) and fine-tuning on a large set of tasks including pos tagging, dependency tagging, NER, GLUE/XNLI and extractive QA. Results show that while BERT often outperforms Pixel on English tasks and latin scripts, Pixel can transfer better to previously unseen scripts. They also conduct analysis on robustness to low-level orthographic attacks and code-switching.\n", "summary_of_the_review": "Recommending weak accept (open to changing this after discussions) because the paper proposes an interesting approach to masked language modeling with a comprehensive set of experiments, but there is a slight lack of clarity in a few places. ", "strength_and_weaknesses": "Strengths:\n1. This paper takes on a new and interesting approach to a very well-established masked language model pre-training/fine-tuning paradigm. It is quite an interesting and refreshing setup.\n2. The paper includes an extensive set of experiments, covering POS tagging, dependency parsing, extractive QA, GLUE, XNLI, NER.\n3. Being able to non-trivially handle new scripts from only fine-tuning data is an impressive feature of the system.\n\nWeaknesses:\n1. It would have been really interesting to see the byte-based vocabulary baseline (something like ByT5). Since most of the experiments involve shorter sequences (right?), is length of the resulting byte-based sequences a concern? In some cases where the vocabulary clearly has very poor coverage, comparing against BERT seems like a slightly unfair comparison with BERT mostly fine-tuning on a sequence of unks. \n2. The discussion on patch span masking is a bit unclear. How was the resolution of the rendered text chosen? Within the same height and sequence length, how many words tend to end up in a patch? Or do patches often contain subwords? I may have missed these details but it seems relevant for understanding how things work. Were there any experiments that varied these parameters to clarify the decisions? A figure illustrating the approach with an example might also be helpful.\n3. The robustness experiments currently lack decided conclusions/takeaways. It doesn’t seem like Pixel is firmly more robust than BERT. Also, the orthographic attack graphs might need error bars?\n4. What if Pixel is provided inputs that are rendered vastly differently during fine-tuning? It would fail to generalize right? Or would the approach be to run OCR and get the text, render it with the Pixel text renderer and then predict? It might be worth adding a discussion that goes over some of these points, and more generally delves into design decisions that are crucial.\n\nNit: maybe the paper title should be masked language modeling with pixels. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JOWlxyxa4G", "reviewer": "Reviewer_2dJC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "## Overview\nIn this paper, PIXEL, a pixel-based encoder of language is introduced to circumvent the vocabulary bottleneck problem. PIXEL is built on ViT-MAE and is pretrained on the same corpus as BERT. Specifically, the authors proposed to render text onto blank images and patchify raw pixels into image tokens and masking 25% of the tokens. Following He et al. (2022), the author then trained PIXEL by using a decoder to reconstruct the masked pixels of the input images. The authors evaluated PIXEL on syntactic and semantic tasks in typologically diverse languages and demonstrated its robustness to orthographic attacks and linguistic code-switching. Although the downstream performance is falling short comparing with BERT and mBERT, the proposed method shows its novelty and potential by providing a new view of tackling the vocabulary bottleneck problem in both mono-lingual and multi-lingual setting. \n\n", "summary_of_the_review": "Although it might be not that useful, I like this idea and it gives a new perspective  for text representation", "strength_and_weaknesses": "## Strength\n- Avoid vocabulary bottleneck problem.\n- Show more robustness to orthographic attacks and code-switching\n- Perform better on non-Latin languages on syntactic tasks and semantic tasks comparing to Bert.\n- Demonstrate the possibility of pixel-based language model without injecting a priori glyph knowledge.\n\nWeaknesses\n- Average performance on downstream tasks still lags far behind Bert and mBert.\n- Data preprocessing (text rendering) is tricky and laborious and will cause inevitable latency in real application.\n- The generated text pixels during pretraining is blurred and fuzzy which makes the model learning less explainable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%