Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer
Authors: Elizabeth Salesky
Neha Verma
Philipp Koehn
Matt Post
Venue: EMNLP 2023 Main (2023)
Keywords: machine translation
pixel representations
multilinguality
cross-lingual transfer
unseen scripts
TL;DR: We use pixel representations to remove the vocabulary bottleneck for multilingual translation models, resulting in more effective and efficient cross-lingual transfer within and across scripts.

Abstract: We introduce and demonstrate how to effectively train multilingual machine translation models with pixel representations. We experiment with two different data settings with a variety of language and script coverage, demonstrating improved performance compared to subword embeddings. We explore various properties of pixel representations such as parameter sharing within and across scripts to better understand where they lead to positive transfer. We observe that these properties not only enable seamless cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. We hope this work contributes to more extensible multilingual models for all languages and scripts.

Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer
Elizabeth Salesky
Neha Verma
Philipp Koehn'
Matt ,,
Johns Hopkins University
H Human Language Technology Center of Excellence M Microsoft
esaleskyējhu. edu
Abstract
C' est une phruse.
33 177- :
We introduce and demonstrate how to effec- tively train multilingual machine translation models with pixel representations. We exper- iment with two different data settings with variety of language and script coverage, demon- strating improved performance compared to subword embeddings. We explore various prop- erties of pixel representations such as parameter sharing within and across scripts to better un- derstand where they lead to positive transfer. We observe that these properties not only able seamless loossllnguul transfer to unseen scripts, but make pixel representations more data-efficien than alternatives such as vocabu- lary expansion. We hope this work contributes more extensible multilingual models for all languages and scripts.
This sentence.
Embeddings are...
while pixels
Disjoint
Shared across scripts
by script
subward umbeeddings
pixels
Fixed vocabularies
Dynamic vocabularies
enabedking matrix allocation
Figure 1: Embedding matrices disjoint parameter allocations by script, leading to a vocabulary bottleneck. Pixel representations however share parameters across scripts and are not limited to discrete vocabulary.
In this work we demonstrate how to effectively parameterize and train multilingual translation models with pixel representations, leading im- provements of up to 9 BLEU on two multilingual datasets with diverse language and script coverage. We explore various properties of pixel representa- tions order to understand their potential bene- fits and limitations, including positive transfer and representational similarity between languages, pa- rameter sharing, and frequency based relationships. Finally, show that not only can pixel represen- tations be finetuned cross-lingually or to unseen scripts, but can do so more data-efficiently than alternatives such as vocabulary expansion, with significant improvements for unseen scripts.
1 Introduction
Multilingual model vocabularies are finite and typ- ically smaller than the possible of Unicode characters, inherently leaving some languages and scripts under-represented. As coverage increases, parameter allocation to each language decreases, resulting in a trade-off between capability, capacity, and coverage. Recent work on pixel representations (Salesky et al., 2021; Rust et al., 2023) provides an appealing alternative to past approaches, because they do not have discrete model vocabulary fi- nite embedding matrix, and can represent all scripts with complete parameter sharing.
2 Our approach
Recent work (Rust et al., 2023) has also shown that paxel-based models can be directly fine- tuned across scripts without vocabulary extensions, adapters, or transliteration. However, pixel rep- resentations have previously only been trained or finetuned on individual languages at time, rather than multilingually. This leaves unanswered ques- tions about the effects of multilingual co-training, such as whether similar scripts will interfere with or boost performance, or if architectural changes will needed given the larger input space.
Covering the larger character sets multilingual models commonly results in significant parameter increases in the embedding matrix and softmax, cre- ating vocabulary bottleneck. While sampling data by language balance vocabularies is common for large-scale multilingual systems (Fan et al., 2021), sampling may cause common vocabulary be out- of-vocabulary (OOV) for languages with longer-tail character distributions like Chinese (NLLB Team
There are 143,698 Unicode character codepoints as of v13.0.

---

al., 2022).² One alternative to move byte- based representations, which combats exploding model parameters by reducing the set of embed- dings to 256. However, this approach increases sequence lengths up to 12x compared to charac- ters, determined by the script's Unicode encoding, making optimal batch sizes prohibitively large and slow for our computational resources.
C' est une phrase.
unicodo atring
C'est une phrase.
render to image
C'e est u ue e p phr ras se.
tokenize
Rendering text to images bypasses many of the vocabulary challenges posed by multilingual mod- eling.
Conv2D
conv. block
Linear Projection
representations have the advan- tage of no predetermined static vocabularies, no ex- ploding embedding matrix parameters or sequence lengths, and complete parameter sharing across similar word forms at sub-character level regard- less of the underlying Unicode or byte structure. Below we present the technical details of our approach and comparisons before proceeding to experimental settings and results.
1X512
flat vectors
Figure 2: Encoding text with pixels: text is rendered to images by sentence. Image tokens are created by over- lapping sliding windows fixed height (h), width (w), and stride (s). Convolutional layer output is projected to flat vectors for subsequent Transformer layers.
compared the umion of subword vocabularies con- structed per-language to a jointly-trained subword vocabulary of the same total size. Individual vocab- ularies were of size 5k," 6 and scaled equivalently for joint vocabularies, e.g. 35k for 7 source languages. The two constructions did not result in significant differences in downstream performances in bal- anced or imbalanced data settings so we present only joint vocabulary results in the main text, as this approach seales more casily to 59 languages. Results for both constructions are shown in App. G. We use separate source and target vocabularies and share target vocabularies between subword and pixel models in order to isolate the source represen- tation change. Vocabulary sizes for all models and datasets are shown in Table in App. B.
2.1 Encoding text with pixels
Figure 2 demonstrates the rendering process and resulting Transformer inputs. We render text using the PangoCairo library following Rust et al. (2023) with font size of 10pt at 120 DPI. We tokenize sentence-level images into fixed-size image tokens with h=24, w=21, and stride s=12, which results in ~3 Latin characters per token. The height was chosen to fit the wide variety of scripts and diacritics in our experimental data with a fixed font size. We use the Google Noto Sans fonts collection which covers the majority of Unicode codepoints. Further discussion rendering parameter choices is found in App. C. No preprocessing applied before rendering. We train many-to-ond multilingual models with pixel representations on the source side, and generate discrete subword tokens as the target as below.
2.3 Model architecture
Our core architecture follows Salesky et al. (2021) and combines convolutional block? which pro- cesses image tokens and produces flattened vec- tors (Figure 2) with a Transformer encoder-decoder model. Convolutional layers use one color channel and a 3 kernel with stride of Our conven- tional text models share the same Transformer ar- chitecture and replace the convolutional block with traditional embedding matrix of size V x 512. Our base models are Transformers with 6 en- coder and 6 decoder layers each, with hidden units of dim 512, feed-forward layers of dim 1024, and
2.2 Traditional subword tokenization
We generated all subword vocabularies using Sen- tencePiece unigramLM (Kudo, 2018; Kudo and Richardson, 2018). In exploratory experiments,
Forr example, the NL.LB model vocahulary does not include the common characters in 'mother' in Chinese, 14.4.
"https://doc" gtk. org/PangoCairo
'PangoCairo provides greater flexibility than alternatives such as PyGame, used in previous work, by supporting fallback fonts the character level.
necessary not only for code-mixing but to support common occurrences such as non-transliterated entities within non-Latin scripts.
Baaed on uning for the same dataset Salesky et al. (2021). A 2D convolutional layer followed hy 2D batch normaliza- tion, a RcL.U layer, and lincar projection to x 512.
"See https: //n////onttttttt/e thub. io/overviee veew for the Noto fonts and their Unicode coverage.

---

28
26
4096
25
25
2048
24
24
1024
Enc. layers
23
22
9
12
21
21
15
20
20
19
19
20M
40M
60M
80M
100M
120M
140M
20M
40M
60M
80M
100M
120M
140M
model
model params
(a) Visual representations
(b) Subword representations
Figure 3: Performance across different model capacities, varying encoder depth and/or width (TED-7).
4 heads. We train our models with the Adam op- timizer (Kingma and Ba, 2015) with linear warm- up, learning rate 5e-4, dropout of 0.1, and label smoothing 0.2. We train with temperature sam- settings
pixel representations and explore any necessary architectural changes in the multilingual setting. We then scale up using the larger 59 language pair TED talk corpus from Qi al. (2018), TED-59. all cases, our models are many-to-one multilingual translation models with English as the target. We list the languages in each corpus with the number of training examples in App. A. Results for all datasets are shown in Table 1.
pling 1.5 in language-imbalanee (Arivazhagan et al., 2019; Shaham et al., 2023). We use batches of 160k tokens, and train until perfor- mance on a held-out validation set fails to improve for ten validations. Trained models and scripts to replicate them will be released upon publication." Reparameterizing model capacity with deeper encoders and shallower decoders has been shown to beneficial particularly for large multilingual vo- cabularies and/or smaller granularity inputs such as characters (Cherry et al., 20 Kasai et al., 2021; Kong et al., 2021; Xu et al., 2021; 2021). Replacing the source embedding matrix with visual representations frees parameters which may be re-allocated elsewhere within the model. As expand language coverage with pixel-based representations, not clear priori whether and where additional capacity may needed to seale performance compared to models for individual languages or traditional text models. We experi- ment with different ways to add and allocate model capacity with both pixel and text inputs, with re- sults presented in 3.1.
3.1 Model capacity: wider or deeper?
Increasing language coverage often requires in- creased model capacity. We find that the small base architecture from Salesky et al. (2021) is un- stable and may not converge when trained multi- lingually without additional capacity or batch size. For TED-7, multilingual source embeddings ac- count for 33% of the total parameters of the best subword model. 9 Without a source embedding ma- trix, despite the additional convolutional block, a pixel model with the same Transformer architecture as subword model would be 1 17M parameters (or 38%) smaller, seen in Figure and may thus require different parameterization and result in different scaling behavior.
1; Berard et al.,
We investigate both the impact of reparameteriz- ing the baseline model, as well as increasing capac- ity through greater encoder depth and/or width. We first find that shifting from equal depth encoder- decoder model to deep encoder and shallow de- coder with the same number of parameters pro- vides consistent improvements; for example, mov- ing from 6 TED-7 datasct from 18.5 to 21.3 BLEU (+2.8). We maintain a shallow layer decoder while varying the encoder through the remainder of this section.
3 Multilingual translation with pixels
We experiment with two datasets to investigate the performance and properties of multilingual pixel representations for machine translation. We perform initial experiments with the balanced 7 lan- guage pair multi-target TED data (Duh, 2018) used by Salesky al. (2021), which will refer as TED-7, to compare performance to prior work with
improves performance on the
TTED-7 uses source subword vncabulary of 35k; this dispar- ity would continue to increase with larger vocabulary sizes.

---

TED-7
TED-59
Source reps.
BLEU
chrF
COMET
BLEU
chrF
COMET
25.7
48.8
77.3
23.8
45.5
73.3
PIXEL
26.2
49.5
77.9
28.4
50.1
77.2
Table 1: Model performance across datasets test. Models chosen by perplexity on held-out validation sets. Metric scores are averaged across all languages in the dataset; App. E shows results for individual language pairs.
pixels hpo
1.00
BLEEU
0.75
BLEU
0.25
0.00
ja
arde fr
bn
ta
Languages
Languages
Languages
TED-7
TED-7
(b) TED-59
Figure 4: Improvement with multilingual models over models for each lang. pair.
Figure 5: Improvement per language with pixel representations
compared to subwords on multilingual TED-7 and TED-59 datasets.
With equal number of model parameters, in- creasing depth is more impactful than width, as seen in Figure 3a. Increasing width provides con- sistent improvements all model sizes, while more significantly increasing overall parameters. With 12-3 layers and 2048 FF width, a pixel-based model has an equivalent number of parameters to the best subword model (~55M) while able to con- tinue improving with scale. The best pixel models for this dataset use 12-3 layers and FF width 4096. Continuing increase depth and overall size has diminishing returns. Pixel models also appear more robust overparameterization where text models degrade more quickly, as seen in Figure 3b.
curse of multilinguulltty (Conneau et al., 2020)). When compare our multilingual models individual models for the same language pairs with TED-7, we see that all languages improve through multilingual training with pixel representations. while this not the case for subword- waasddd--aeddd mod- els, where two language pairs degrade (Figure 4). Improvements are greatest for those language pairs (ja, ko, zh) where individual models performed than BPE in Salesky et al. (2021). Improve- ments could be due boosts from languages with similar scripts (zh and ja, or and de) or simply an increase in total training data: we investigate this in 4.J 1 for TED-59 where we have more languages to study. Notably, improvements come without interference for pixel models here. Comparing multilingual pixel and BPE models, we see small but consistent improvements on TED-7 (Figure 5),
Is the optimal parameterization determined by the granularity of pixel inputs, the amount of train- ing data, or the multilinguality of the task? To see, reparameterize the models for individual language pairs from Salesky al. (2021) at both the small and large data sizes (shown in App. F). We find that performance would have decreased in both cases, suggesting this is more likely due to the multilingual task, not the amount of data or pixel representations inherently.
The TED-7 setting has relatively balanced data across all languages and scripts and at least 150k examples per pair, which is a reasonable baseline but unrealistic in the context of typical multilingual translation settings. We turn the TED-59 dataset for increased language coverage with imbalanced training data and script representation for more realistic setting to see if our improvements hold or interference emerges. Here we see larger im- provements of up to BLEU compared to BPE for most language pairs, and some degradation for 2 pairs whose scripts have only ~5k training exam- ples across all languages, highlighted in Figure 5. Given the large and imbalanced nature of this
For the larger TED-59 dataset (1.2M-5.IM), we use the same architecture for TED-7. Exact model configurations for each dataset and represen- tation scheme are listed together in App. B.
3.2 Language coverage and imbalanced data
Including additional languages can sometimes in- terfere with rather than improve performance ('the

---

8
4 BLEU
2
0
p=0.44
p=0.35
p=0.70
-2
-2
-2
10k
50k
100k
200k
100k
250k
500k
750k
100k 250k
500k
750k
Sentences per language
Sentences per family
Sentences per script
Figure 6: Performance improvements with pixel representations are most strongly correlated with the total amount data for a language's script compared to language language family. Data size per language listed App.
Sre
Script
Sents
PIXEL
BPE
Aharoni.
A
25.3 average BLEU: our many-to-one pixel model
az
Latin
S946
16.6
12.5
11.2
+5.4
improves on this by +3.1 BLEU. They do not report results per language for further comparison.
be
Cyrillic
4509
28.5
19.2
18.3
+10.2
gl
Latin
10017
36.5
29.7
28.6
+7.9
Latin
61470
33.7
27.4
26.8
+6.9
Properties of multilingual pixel models
avg:
28.8
22.2
21.2
+7.6
4.1
Positive transfer across languages
Arabic
214111
29.8
26.1
25,9
+3,9
de
Latin
167888
36.1
30,0
28.9
+7.2
We look the relationship between data represen- tation for each source language, family, script and performance find the greatest contributors im- provements with pixel representations on TED-59. The amount data for a given pair is only weakly related to performance for both pixel and subword representations (p50.3,, 0.05), while language family and script representation is moderately cor- related (p-0.5 - 0.6, < 0.001) suggesting some positive transfer across languages and scripts for both approaches. However, looking each factor's relationship to performance improvement rather than raw scores better reflects those responsible for the difference. As shown Figure 6, the amount of data for a given script is strongly correlated with ABLEU, (p-0.70, p < 0.001), while family is moderately correlated (0.35) and data for indi- vidual language pairs has no clear relationship. We conclude that pixels enable more effective cross- lingual transfer between languages with the same script, and a lesser degree family, than joint sub- word vocabularies. We hypothesize that we would see similar improvements for Bengali and Tamil with at least 10k examples for their scripts.
Hebrew
211819
35.3
30.7
30.2
+5.1
it Latin
204503
38.5
32.3
32.4
+6.1
[HR]
avg:
34,9
29.4
+5.6
Table 2: Results for 4 high-resouree (HR) and low- resource (IR) language pairs used in previous work.
dataset, previous work has commonly reported non- aggregated performance for subset of language pairs only (4 low-resource and 4 high-resource) with varied scripts and degrees of relatedness. Compared to the best previous results those pairs (Aharoni et al., 2019), our subword baselines improve slightly: +1 BLEU on the LR pairs and +0.4 on the HR. With pixel representations, our models improve significantly, +7.6 on the LR pairs and +5.6 on the HR pairs, as shown in Table Low-resource languages with well-represented scripts shared with other languages show larger improvement than the overall mean of +4.6; most dramatically, Belarusian (be) improves by >50% 10.2 BLEU despite having only 4509 training instances through positive transfer from the >600k sentence pairs Cyrillic TED-59 (discussed fur- ther 4.1.. Jin and Xiong (2022) presented the strongest previous performance on the full TED-59 dataset with many-to many multilingual model with language-awar multi-head attention, with
4.1.1 Clustering by language and script
To better understand how pixel representations pattern by language and script compare our model subword embeddings and our pixel repre- sentations. Using the validation set as input, we compute sentence-level vectors by mean- mean--oolingg over token embeddings for each sentence for the
WhTheir many-to- to-y-to-mee many model is trained on 2x as many sen- tences as the models presented here by reversing the dataset.

---

Donk
(a) PIXEL, by script
(b) SUBWORD, by script
(c) PIXEL, by lang/family
(d) SUBWORD, by lang/family
Figure 7: Clustering shows more representational similarity within scripts and actoss languages with pixel represen- tations than with disjoint subword embeddings in the TED-59 dataset. Individual languages from the same family are shown with different shades the same color in items (c) and (d).
subword model, or over the linearly projected vec- tors of the same dimension from the convolutional block in the pixel model. We visualize these repre- sentations using t-SNE clustering (van der Maaten and Hinton, 2008), in Figure 7 for TED-59 and in App. for the smaller TED-7.
Canvid
Pixel representations cluster neatly by script (7a), reflecting the strong ability to share information between languages of the same script discussed in 4.1. Subword embeddings do not cluster as strongly by script despite shared subwords, with many separate clusters for e.g. Latin script lan- guages (7b). We observe that subword embed- dings cluster more tightly by language and family (7d), with less representational overlap between lan- guages than we see with pixels (7c). However, the visual model still reflects some similarities within families both within and across scripts. For ex- ample, in the large Latin-script cluster in 7c, all Uralic languages appear within close proximity of each other, as do Austronesian, and some overlap exists between Cyrillic and Latin representations in 7a which likely reflects Slavic family similar- ities rather than visually similar characters given sentence-level vectors.
Figure 8: Pixel representations result in similar rep- resentations for partial lexical matches due to visual similarity and parameter sharing the pixel level.
words e.g. with and without diacritics-witt the TED-59 subword vocabulary, the Arabic forms كاب and كنu for "book" have disjoint subword decom- positions and so do not share embeddings, whereas the pixel representations are highly similar; as visu- alized in Figure 8, the convolutional layer feature activations remain highly similar despite the in- serted diacritics. If pixel-based model observes partial lexical matches such as "ktb" and "kitab" in training, parameters for both will be updated by backpropagation to the shared pixel values; we hypothesize that this contributes to the increased transfer across languages with the same script and performance improvements. Future work may in- vestigate whether this property leads to more com- positional representations.
4.2 Complete parameter sharing
With traditional model vocabularies, parameters are not shared between embeddings; only 3% of embeddings are updated per batch on average"! for TED-59 without redistribution techniques such as label smoothing. On the other hand, 100% of the pixel model representation block parameters are updated every batch due to parameter sharing at the pixel level. Pixel representations have direct ac- cess to token sub-components, whereas subwords do not, leading to more similar representations for
4.3 Reduced frequency- representation degeneration
Previous work has shown that embeddings can suf- fer from a frequency- representation degener- ation problem, where infrequent and unseen words cluster together in embedding space due to lim- ited parameter updates during training (Gao et al., 2019). However, as pixel models share parameters at the pixel level, all representations are updated to some degree each batch regardless of subword-
1| Heavvll dependent on language coverage, sampling, vocab- ulary, and batch size. This number reflects 64k souree vocahulary and large batch size of 160k tokens.

---

40
Representations
visrep
30
bpe (ext)
bpe
BLEU
20
(a) TED-7. pixel
(b) TED-7. subword
10
Figure SVD plots of source representations show traditional embeddings cluster infrequent subwords to- gether more tightly than pixels.
0
10k
50k
150k
Num, samples
level text frequency. Therefore, the low-frequency degradation effect should reduce in pixel models and rare words may not cluster as strongly.
Figure 10: Data-efficiency cross-lingual transfer. Models with pixel-baset representations adapt more efficiently and effectively to new scripts than with tradi- tional text representations (shown here: Hebrew).
We examine this phenomenon by comparing the source embeddings from the subword model against representations from the pixel model on TED-7. We obtain a comparable set of represen- tations from the pixel model by rendering each subword in the TED-7 source vocabulary and mean- pooling the output of the convolutional block for all resulting resulting visual token(s).
Script scen?
Unigram Coverage
Bigram Coverage
Trigram Coverage
Gain
Gain
BPE
BPE
m pl
96%
91%
84%
+11%
+11%
95%
88%
73%
+13%
+14%
fa
/
99%
79%
66%
+18%
+20%
vi /
86%
66%
41%
+22%
+21%
be x
23%
5%
15
+30%
+4700%
We plot these embeddings using 2-D singular value decomposition, and color each point accord- ing to the log-frequency of its corresponding sub- word in Figure 9. We plot visual embeddings, ex- cluding 1% of outliers for improved readability (and include the full plot in App. I). We see that in the text model, there is both clear frequency bias and and a cluster of oow-frequency embed- dings. In the pixel model, though we see some frequency bias among embeddings, the distribution low-frequency embeddings improved.
Table 3: Script coverage in pretraining measured at the
level of character r-grams. Improvements with pixel representations are averaged across all resource settings.
tensions, but enable transfer more data-efficiently requiring fewer examples to achieve good perfor- mance.
To evaluate the data-efficiency of cross-lingual transfer, we adapt our multilingual models to lan- guage pairs with five new source languages, cach with different degrees script coverage to those observed in pretraining as quantified in Table 3: Romanian, Polish, Farsi, Vietnamese, and Hebrew. We randomly sample 10k, 50k, and 150k (~all) sentences from the multi-target TED dataset used for TED-7 for each new language pair and fine- tune our TED-7 models on the training data for each pair individually for up to 30 epochs, with early stopping if there are no improvements on the held-out validation sets for 5 epochs. We use the TED-7 models because they do not cover these languages in pretraining: we note that the overall performance the original task is similar for pixel and subword models. In addition the pixel and subword models, also compare subword mod- els with vocabulary expansion, where the source embedding matrix is extended to include BPE in- ventories of size 5k trained for each new language,
5 Data-efficient cross-lingual transfer
It has been shown that using pretrained multilin- gual models for cross-lingual transfer can provide significant performance improvements, particularly when the target language under-resourced. How- ever, adapting models to unseen scripts with no lexical coverage in the original model typically requires techniques such as expanding the embed- ding matrix to include new vocabulary (Wang et al., 2019b) or language-specific adapters (Houlsby et al., 2019; Pfeiffer et al., 2020). In contrast, mod- els with pixel representations can be finetuned di- rectly on new languages and scripts without requir- ing any architectural changes (Rust et al., 2023). We hypothesize that the model properties discussed in $ will not only allow transfer without model ex-

---

for which embeddings are randomly initialized. Whether model vocabularies cover a particular script is lypically described as binary, but even with observed scripts new languages introduce un- seen character sequences and diacritics which will not be appropriately represented. We observe that for Unicode-based models, transfer capability is strongly reflected in lexical coverage; vocabulary expansion improves performance slightly for lan- guages with higher n-gram coverage, and signifi- cantly for Hebrew with minimal coverage, particu- larly with more data to train new language-specific embeddings, as seen in Figure 10. However, pixel representations enable models to perform better still than vocabulary expansion, particularly with less data. We believe this is because with complete parameter sharing across all scripts, all parameters for new languages are more strongly initialized. This direction may lead to more data-efficient cross- lingual transfer, particularly for under-resoureed languages and tasks.
lated languages across scripts, though they may also introduce collisions which can negatively af- feet performance. Post-hoc vocabulary expansion (Wang et al., 2019b; Moon and Okazaki, 2020) or language adapters (Houlsby et al., 2019; Pfeif- fer et al., 2020) to increase vocabulary coverage have also been shown to be very effective. Re- cently, pixel representations have been proposed vocabulary-free alternative (Salesky et al., 2021; Rust et al., 2023), though not trained yet multilin- gually. We refer readers to the BigScience survey for greater discussion (Mielke et al., 2021).
7 Conclusions
We introduce and demonstrate how to effectively train multilingual pixel representations for machine translation. We experiment with two different data scales with variety of language and script cov- erage, demonstrating improved performance com- pared to the traditional subword approach. We analyze various properties of pixel representations to better understand where they may provide po- tential benefits and the impact of different scripts and data representation. We observe that these properties not only enable cross-lingual transfer to unseen scripts, but make pixel representations more data-efficient than alternatives such as vocabulary expansion. We hope this work contributes to more extensible multilingual models for all languages and scripts.
6 Related Work
Previous work has shown allocating additional en- coder capacity to be beneficial for smaller granu- larity inputs, both for characters and bytes (Cherry et al., 2018; Xue et al., 2022b) and other modalities (He et al., 2021; Zhang et al., 2017). Deep encoders and shallow decoders have been used to improve model efficiency and latency with subword inputs (Kim et al., 2019; Kasai et al., 2021; Kong al., 2021), and deeper and narrower encoders have been shown to scale more effectively (Tay et al., 2022: Xue et al., 2022a).
8 Limitations
Our multilingual experiments are only many-to- one thus far, and apply visual representations to the source languages only. Whether the dynam- ics would change with multiple target languages is not yet known. Though we do experiment with multiple resource scales up to ~5M sentences settings remain limited in scale and domain com- pared to large-scale industry models and it remains to be seen how this approach would fare in other settings. At very low-resource settings with fewer than 10k examples for a given script, our approach may perform worse than traditional subword em- beddings. We observe that pixel models are in some settings slower to converge than subword equiva- lents, which we cautiously attribute to sub-optimal hyperparameters. Though the compute resources required for training models are similar to tradi- tional text representations, significantly more disk space is required to save rendered text compared to
Significant prior work has been devoted to broader and more effective language coverage, through full Unicode character coverage and down- sampling (Clark et al., 2022), clustered vocabu- laries for efficient modeling of large vocabular- ies (Chung et al., 2020; Liang et al., 2023), byte- level modeling (Gillick et al., 2016; Xue et al., 2022b), bytes in conjunction with BPE to com- bat data sparsity and memory issues (BBPE: Rad- ford et al., 2019; Wang et al., 2019a) or byte- fallback (Xue et al., 2022b). Mapping characters to smaller set of common representations across scripts through transliteration (Amrhein and Sen- nrich, 2020; Purkayastha al., 2023) or grapheme- to-phoneme systems (Sun et al., 2022; Gheini and May, 2019) have also been shown beneficial for multilingual and cross- lingual transfer for re-

---

raw text, which may be necessary if pre-computing batches without rendering on-the-fly and may limit efficiency in larger-scale settings. Sealability to longer text has not yet been investigated.
Chantal Amrhein and Rico Sennrich. 2020. On Roman- ization for model transfer between scripts in neural machine translation. In Findings of the Asvociation for Computational Linguistics: EMNLP 2020, pages 2461-2469, Online. Association for Computational Linguisties.
9 Ethics Statement
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges.
The aim of this work is to reduce the vocabulary bottleneck which disproportionately affects low- resource languages as they are less likely be ap- propriately represented in traditional discrete mul- tilingual model vocabularies. Alternatives such as byte-level tokenization potentially increase rather than decrease the disparity between scripts, as a sin- gle character may be represented as up to 12 bytes in e.g. Telugu, whereas Latin scripts are typically 1:1 characters.bytes (Ahia al., 2023). We show the sequence lengths resulting from byte, charac- ter, BPE, and pixel 'tokenization' on TED-59 in Figure 11, App. D; of the alternatives to BPE tok- enization, pixel representations result in the most similar sequence lengths and lowest variance across languages and scripts.
Alexandre Berard, Dain Lee, Stephane Clinchant, Kweonwoo Jung, and Vassilina Nikoulina. 2021. Efficient inference for multilingual neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 8563-8583, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Colin Cherry, George Foster. Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting character- based neural machine translation with capacity and compression. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Procexsing, pages 4295-4305, Brussels, Belgium. Association for Computational Linguistics.
In application settings, substituting visually sim- ilar characters such as 'O' for 'O' can be used to cireumvent lexical filtering as used for e.g. spam fil- tering, hate speech detection, or censorship. Pixel representations may make these substitutions less effective which may be beneficial or harmful de- pending on the setting.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536 4546, Online. Association for Computational Linguisties.
Acknowledgments
We are grateful to Team PIXEL (Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Miryam de Lhoneux, Desmond Elliott), Carlos Aguirre, An- tonis Anastasopoulos, and Chenlei Sei for helpful discussions. Elizabeth Salesky is supported by the Apple Scholars in AI/MI. fellowship.
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wieting. 2022. Canine: Pre-training an efficient tokemization-free encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- linsulln representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
References
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North
American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Kevin Duh. 2018. The multitarget TED talks task.
http://www.co http:///wwwcs.h..c.oo htt/ee///kkviddu//all Mettitarget-tedtallke
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yu- lia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language mod- els.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary. Naman Goyal, Tom Birch, Vi- taliy Liptchinsky, Sergey Edunov, Edouard Grave,

---

Michael Auli, and Armand Joulin. 2021. Be- yond english english-centric multilingual machine transla- tion. Journal of Machine Learning Research.
of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1613-1624, Online. Association for Computational Linguistics.
Jun Gao, Di He, Xu Tan, Tao Qin. Liwei Wang, and Tieyan Liu. 2019. Representation degeneration prob- lem in training natural language generation models. In International Conference on Learning Representa- fions.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 66-75, Melbourne, Australia. Association for Computational Linguisties.
Mozhdeh Gheini and Jonathan May. 2019. A univer- sal parent model for low-resource neural machine translation transfer.
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok- enizer and detokenizer for neural text processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Dan Gillick, Cliff Brunk. Oriol Vinyals, and Amarnag Subramanya. 2016. Multilingual language process- ing from bytes. In Proceedings of the 2016 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. pages 1296-1306, 306, San Diego, California. Association for Computational Linguis-
tics.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. XIm-v: Overcom- ing the vocabulary bottleneck in multilingual masked language models.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked au- toencoders are scalable vision learners.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. efffeietr-efficientt transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Jonas F. Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott. 2023. Text rendering strategies for pixel language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Longuage Processing. Association for Computational Linguistics.
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Sam- son Tan. 2021. Between words and characters: A brief history of open-vocabulary modeling and tok- enization in nlp. ArXiv, abs/21 12.
Renren Jin and Deyi Xiong. 2022. Informative language representation learning for massively multilingual neural machine translation. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 5158-5174 Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Sangwhan Moon and Naoaki Okazaki. 2020. Patch- BERT: Just-in in-t-in-emme, oft-of-vocabulant patching. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7846-7852 Online. Association for Computa- tional Linguistics.
Jungo Kasai, Nikolaos Pappas, Hao Peng. James Cross, and Noah Smith. 2021. Deep encoder, shallow decoder: Reevaluating non-autoregressive machine translation. In International Conference on Learning Representations.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafilld Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood. Bapi Akula, Loic Bar- rault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman. Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers. Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation. arXiv preprint.
Young Jin Kim, Marcin Lunezys-Dowmunt Hany Has- san, Alham Fikri Aji, Kenneth Heafield. Roman Grundkiewicz, and Nikolay Bogoychev. 2019. From research to production and back: Ludicrously fast neural machine translation. In Proceedings of the 3rd Workshop Neural Generation and Transla- tion, pages 280-288, Hong Kong, Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In International Conference Learning Representations.
Xiang Kong, Adithya Renduchintala, James Cross, Yuqing Tang, Jiatao Gu, and Xian Li. 2021. Mul- tilingual neural machine translation with deep en- coder and multiple shallow decoders. In Proceedings
Jonas Pfeiffer. Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter-Based

---

Framework for Multi-Tass Cross- LingslLingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, Online. Association for Computa- tional Linguistics.
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019b. Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 316-327, Hong Kong, China. Association for Computational Lin- guistics.
Sukannya Purkayastha, Sebastian Ruder, Jonas Pfeif- fer, Iryna Gurevych, and Ivan Vulić. 2023. Romanization-basee -based large-scale adaptation of multi- lingual language models.
Hongfei Xu, Josef van Genabith, Qiuhui Liu, and Deyi Xiong. 2021. Probing word translations in the trans- former and trading decoder for encoder layers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 74-85, Online. Association for Computational Linguistics.
Ye Qi, Devendra Sachan, Matthieu Felix. Sarguna Pad- manabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neu- ral machine translation? In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pages 529-535, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
Fuzhao Xue, Jianghai Chen, Aixin Sun, Xiaozhe Ren, Zangwei Zheng, Xiaoxin He, Xin Jiang, and Yang You. 2022a. Deeper wider: revisit of trans- former configuration.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022b. ByT5: Towards a token- free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliza- beth Salesky, Miryam de Lhoneux, and Desmond El- liott. 2023. Language modelling with pixels. /CLR.
Yu Zhang, William Chan, and Navdeep Jaitly. 2017. Very deep convolutional networks for end-to-end speech recognition. In 2017 IEEE international con- ference on acoustics, speech and signal processing (ICASSP), pages 4845-4849, IEEE.
Elizaheth Salesky, David Euter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 7235-7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Uri Shaham, Maha Elbayad, Vedanuj Goswami, Omer Levy, and Shruti Bhosale. 2023. Causes and cures for interference in multilingual translation.
Simeng Sun, Angela Fan, James Cross, Vishrav Chaud- hary, Chau Tran, Philipp Koehn, and Francisco Guzmán. 2022. Alternative input signals trans- fer in multilingual machine translation. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Pa- pers), pages 5291 -5305, Dublin, Ireland. Association for Computational Linguistics.
Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang. Dani Yogatama, Ashish Vaswani, and Donald Met- zler. 2022. Scale efficiently: Insights from pretrain- ing and finetuning transformers. In International Conference Learning Representations.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-sne. Journal of Machine Learning Research, 276)22559-
Changhan Wang. Kyunghyun Cho, and Jiatao Gu. 2019a. Neural machine translation with level subwords. In AAAI.

---

A List of Languages by Dataset
We list the source languages in each dataset with the number of training examples and language code. All datasets are many-to-one parallel with English as the target language. For TED-7 and TED-59, we use the provided train/dev/test splits, and report results on test using model checkpoints chosen based on dev perplexities.
TED-7
ar Arabic
175k
ja
Japanese
155k
zh
Chinese Total:
170k
de German
153k
Korean
166k
1.2M
fr
French
158k
ru Russian
181k
TED-599
ar
Arabic
214k he Hebrew
212k
pl
Polish
176k
Azerbaijani
6k hi Hindi
19k
pt
Portuguese
52k
be
Belarusian
5k hr Croatian
122k
pt-br
Br. Portuguese
185k
bg
Bulgarian
174k hu Hungarian
147k
Romanian
180k
bn
Bengali
5k
hy Armenian
21k
Russian
208k
bs
Bosnian
6k
id Indonesian
87k
sk
Slovak
61k
calv
it Italian
205k
sl
Slovenian
20k
es
Czech
103k
ja Japanese
204k
sq
Albanian
45k
da
Danish
45k
Georgian
13k
Scrbian
137k
de
German
168k
kk Kazakh
3k
Swedish
57k
el
Greek
134k
ko
Korean
206k
ta
Tamil
6k
co
Esperanto
7k
ku
Kurdish
10k
th
Thai
98k
es
Spanish
196k
It
Lithuanian
42k
tr
Turkish
182k
ct
Estonian
11k
mk
Macedonian
25k uk Ukrainian
108k
eu
Basque
mn
Mongolian
8k
Urdu
fa
Farsi
151k
mr Marathi
10k
Vietnamese
172k
Finnish
24k
Malay
5k
zh
Chinese
6k
fr
French
192k
my
Burmese
21k
zh-en
Chinese, Simplified
200k
fr-ca
Ca. French
20k
nb
Norwegian Bokmål
16k
zh-tw
Chinese, Traditional
203k
gl
Galician
10k
nl
Dutch
184k
Total:
5.1M
B
Model details by dataset
Below we report the details of the best performing model for each dataset and source representation.
Dataset
#Sents
Model V
Vare
Vegt
Emb. dim.
Enc. layers Dec. layers FF width
Attn. heads #Params
TED-7
1.2M
PIXEL.
Ø
10k
512
12
4096
4
87M
TED-7
1.2M BPE
35k
10k
512
6
1024
4
55M
TED-59
5.1M PIXEL.
2
10k
512
12
4096
4
87M
TED-59
5.1M BPE
64k
10k
512
6
2048
8
82M
Table Details of pixel and subword model scale variants.

---

C Detailed discussion of rendering parameter choices
Below we discuss our rendering choices in further detail and provide pointers to the experimentation past work we build from.
Font:
Following past work (Salesky al., 2021; Rust al., 2023) the Noto font family, as it has the widest Unicode coverage within single font or font family known Previous work has used the non-serif font variant: we find slight performance decrease of 5% with NotoSerif on TED-59, and accordingly stick to NotoSans.
Patch size and stride: Salesky et al. (2021) extensively tune font size, window size, and stride for single language pair translation experiments, and find that performance may degrade for some language pairs with font size <10pt. For this reason, we use font size 10pt. While that work found slight differences in optimal window size (15-30) and stride (5-20), found degradation in multilingual performance with uniform window widths and use uniform values for simplicity. Rust et al. (2023) used smaller square windows of 16 16, without any patch overlap (continuous), for English pretraining and cross- lingual finetuning for classification tasks. In our multilingual translation experiments TED-59, we find an average 10% performance decrease without any overlap (stride s = width w). The maximum height of the characters in TED-59 with font size 10pt is 22px, requiring reduced size or truncation to use window size 16pt. A larger window size of 32 fits all characters but increases the proportion of whitespace pixels, and decreases performance by 7%. With window size 24px we were able to fit all characters and diacritics in this dataset, with best overall performance.
Convolutional block: We find an average reduction in performance of 2.8% (28.4 to 27.6 BLEU, sidec = 0.6) without convolutional block, translating directly from pixel values linearly projected 512-dim vectors, and so use convolutional block in all experiments reported here.
Additional rendering strategies: Lotz et al. (2023) compare additional rendering strategies to decrease the pixel input space through structured spacing (bigrams, words) monospace fonts where available, and show improvements both pretraining and cross-lingual transfer to downstream classification tasks, and multilingual QA. It remains be seen how these strategies would affect translation.
Rendering backend: In addition to the character-level fallback capabilities mentioned in the main text ($ 2.1), the PangoCairo renderer is also more efficient than PyGame, with throughput approaching the Rust-based BERT tokenizer without batch processing, as measured in Rust et al. (2023, App. D).

---

D Variance in sequence lengths across tokenizations
Below we show the sequence lengths resulting from byte, character, and BPE tokenization and pixel representations on TED-59. the alternatives to BPE, pixel representations result in the most similar sequence lengths and lowest variance across languages and scripts.
BPE
characters
bytes
pixels
pixels (no overlap)
500
400
Avg. sequence length
300
200
100
Nu
mi mi
my
m
sk
m
Language
Figure 11: Average sequence length with various tokenization schemes compared on TED-59.
E Full results reported by individual language pair
addition to the aggregated metric scores reported the main text, below we report results for each individual language pair with three metrics: BLEU, chrF, and COMET.
Results are organized by dataset. TED-7 results are reported in Table 5, and TED-59 in Table 6.
E.1 Individual language pair results: TED-7
BLEU
Mean
ar
de
fr
ja
ko
TU
zh
PIXEL
26.2
32.1
35.4
37.1
16.0
18.1
26.1
18.9
BPE
25.7
31.4
34.3
36.3
15.7
17.7
25.7
18.8
CHAR
24.5
30.4
32.7
34.3
15.0
16.8
24.3
17.8
chrF
Mean
ar
de
fr
ja
ko
TU
zh
PIXEL
49.5
54.5
57.6
58.4
40.4
42.7
49.6
43.4
48.8
53.2
56.5
57.6
40.5
42.5
48.8
42.7
CHAR
47.5
52.4
55.0
56.0
39.2
41.1
47.6
41.4
COMET
Mean
de
ko
zh
PIXEL
77.9
79.6
79.1
81.6
75.2
76.6
76.5
76.6
BPE
77.3
78.8
77.9
76.4
75.7
75.8
CHAR
76.7
78.8
78.1
80.2
74.3
75.6
75.3
74.5
Table 5: Results on TED-7 evaluation reported by individual language pair.

---

E.2
Individual language pair results: TED-59
BLEU
Mean
be
bg
bn
bs
C5
da
de
el
28.4
29.8
16.6
28.5
39.6
12.7
38.3
30.7
44.8
36.1
38.0
32,9
BPB
26.1
19.2
31.4
25.0
38.3
30.0
33.0
65
et
eu
fa
fi-ca
gl
he
hr
hu
PIXEL
41.8
23.6
21.6
26,9
22.9
40.0
35.0
36.5
35,3
21.9
37.9
26.8
36.2
29.7
30.7
20.8
31.7
by
id
it
ja
ka
kk
ko
ku
1t
mk
mm
mnr
PIXEL
23.7
32.1
38.5
13.2
21.7
11.8
18.1
18.4
27.2
35.3
11.9
11.8
20.1
26.7
32.3
11.8
7.9
16.2
15.4
21.5
30.2
10.8
1118
my
nb
n1
pl
pt
pt-br
ro
ru
sk
s1
PIXEL
26.1
16.2
46.8
36.0
25.7
43.8
45.0
36.3
25.7
33.7
28.7
40.0
21.7
14.1
39.7
30.4
20.8
37.3
38.8
30.0
21.7
27.4
23.0
31.9
81
SV
ur
vi
zh
zh-en
zh-tw
PIXEL
37.3
40.4
7.6
22.5
26.3
30,2
19.1
26.8
17.1
18.5
17.4
BPB
30.4
33.2
7.7
19.8
21.1
24.9
16.8
23.2
14.7
17.3
16.5
chrF
Mean
be
bg
bn
bs
CS
da
de
el
50.1
51.2
38,9
50,8
60.5
32.2
59.9
53.7
64.2
57.7
57.9
53.3
BPE
53.6
48.0
58.6
51.6
53.0
05
et
cu
fa
fr-ca
gl
he
hr
hu
PIXEL
62.7
46.6
44.7
49.6
44.5
60.9
56.9
58.3
55.5
41.9
59.4
49.8
38.3
51.8
52.3
51.1
53.6
hy
id
it
ja
ka
kk
ko
ku
1t
mk
mm
mer
PIXEL
44.8
54,2
59.6
36.9
43.1
33.6
41.4
39.7
50.2
57.8
33.3
32,0
41.1
48.4
54.1
34.9
39.5
28.4
39.3
36.2
44.1
52.1
29.4
31.5
m18
my
nb
pl
pt
pt-br
ro
ru
s1
PIXEL
51.8
38.9
64,9
57.4
48.8
64.3
64,9
57.9
49.2
55.8
51.7
60.0
45.3
36.9
58.8
52.0
43.4
59.0
59.6
52.2
44.6
49.8
46.1
52.5
SV
ur
vi
zh
zh-en
zh-tw
PIXEL
58.9
60.6
25.8
44.8
49.7
52.6
40.2
49,1
37.9
41.2
40.2
BPR
52.3
54.1
28.1
42.7
43.8
47.2
37.4
45.1
36.9
40.3
39.6
COMET
Mean
ar
az
be
bg
bn
bs
C5
da
dc
cl
CO
PIXEL
77.2
77.4
74.2
76.2
82.5
62.9
84.3
80.0
83.7
81.3
81.8
78.5
BPE
66.8
74.3
79.7
78.2
CS
ct
cu
fa
fr-ca
gl
he
hr
hu
PIXEL
83.4
74.6
74,3
77.1
76.2
82.5
82.8
81.9
79.4
70.3
82.8
77.9
79.2
69.8
73.7
70.2
78.4
77.8
76.9
75.8
71.8
77.7
72.4
hy
id
it
ja
ka
kk
ko
ku
It
mk
mm
mr
PIXEL
75.5
81.0
82.3
71.9
71.1
65.1
75.4
63.7
78.4
81.4
68.3
65.3
73.7
76.0
77.4
69.7
69.7
60.5
73.5
61.5
72.3
77.0
66.1
66.2
ms
my
pl
pt
pt-br
no
ru
PIXUL
78.6
73.2
84.0
81.4
76.8
84.5
85.2
82.2
77.0
81.2
78.5
81.9
74.1
71.6
79.1
76.5
71.4
81.1
80.4
76.8
72.6
75.6
73.5
76.2
51
SV
ur
vi
zh
zh-en
zh-tw
PIXEL
82.0
82.5
59.8
75.9
79.8
78.8
71.3
77.8
74.5
73.8
71.5
BPF
76.3
77.3
61.7
74.8
74.6
73.8
69.1
74.4
68.7
72.9
72.1
Table 6: Results TED-59 evaluation reported by individual language pair.

---

F Control experiment: Reparameterized models for individual language pairs
Here we reparameterize the TED models for individual language pairs from Salesky et al. (2021) according to our findings in $ 3.1, shifting encoder decoder layer depth from 6-6 to 12-3 and feed-forward width from 1024 to 2048, while maintaining approximately the same number of parameters (55.3M vs. 56.9M). We see that this reparameterization is not optimal for individual language pairs with less data, or the individual de-en language pair with similar amounts of data to TED-7.
TED
WMT
Model
ar
de
fr
ja
ko
ru
zh
Mean
de
Salesky et al. (2021)
32.1
33.6
36.7
14.4
17.0
25.4
18.3
25.4
32.9
Reparameterized
31.0
33.5
35.6
12.6
15.4
24.1
17.0
24.2
29.4
Table Performance differences by reparameterizing models individual language pairs according to 3.1.
Subword vocabulary constructions
Here we compare different multilingual subword vocabulary constructions for baseline text models on TED-7, with results for bilingual models from Salesky et al. (2021) for comparison.
Model
ar
de
fr
ja
ko
ru
zh
Mean
Bilingual
32.1
33.6
36.7
14.4
17.0
25.4
18.3
25.4
Characters
30.4
32.7
34.3
15.0
16.8
24.3
17.8
24.5
Joint vocab
30.6
33.8
36.3
15.6
17.4
25.1
18.6
25.3
Separate vocab
31.4
34.3
36.3
15.7
17.7
25.7
18.8
25.7
Table
Results different vocabulary constructions for baseline text models on TED-7.
Clustering by language and script: TED-7
Below we show the same t-SNE clustering from $ 4.1.1 for the smaller multi wayl parallel TED-7 validation set. Sentence-level vectors for clustering are creating by mean-pooling token embeddings for both the PIXEL and BPE models. We observe clear clustering by source language in the text model, despite parallel sentences and shared subwords. In the pixel model, observe multiple clusters per language and script, with greater overlap between languages with shared scripts (French and German).
-20
25
de
de
-30
KD
40
-75
2h
2h
-75
-50
-25
25
33
-00
40
-20
20
40
60
TED-7, PIXEL
(b) TED-7, SUBWORD
Figure 12: Clustering shows more representational similarity across languages and scripts with pixel representations than with disjoint subword embeddings.

---

I Full SVD plot of TED-7 pixel model embeddings
Most Frequent
14
0.06
-12
0.05
10
0.04
log(Count)
0.03
0.02
-0.01
-0.0175
-0.155
-0.0125
-00100
0.0075
0.0000
0025
Least Frequent
Figure 13: Full SVD visualization of source-side embeddings from TED-7 pixel model. Only 1% embeddings lie above y 0.03, which was excluded from the main text assist readability.


Reviews:
[{"review_id": "HMkqcEYLLT", "reviewer": "Reviewer_rKto", "summary": "", "questions": "A: 101 On the target side, you decode subword tokens. What is the vocabulary size and construction method for the target subword vocabulary?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review! \n\nQuestion A:  \nWe note on L117-121 that we use the same target vocabulary for both pixel and subword input models and list vocabulary sizes in Table 4. Vocabulary construction is SentencePiece unigramLM (L103) and $V_{tgt}$ is 10k for all datasets (Table 4).  \nWe will add that Table 4 is in Appendix B after it is referenced on L121 and make it clearer that SentencePiece unigramLM is used for all subword vocabularies in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed pixel representations improve the crosslingual transfer between languages sharing a script,\nas well as enabling transfer between historically related but distinct scripts encoded with different unicode codepoints.\nThe proposed method is thus especially beneficial for low-resource languages using non-Latin scripts, a disadvantaged group of languages in NLP.\n\nModels using pixel representations are more robust when scaling the model size.", "reasons_to_reject": "None", "questions_to_authors": "A: 101 On the target side, you decode subword tokens. What is the vocabulary size and construction method for the target subword vocabulary?", "paper_topic_and_main_contributions": "Evaluating the use of pixel input representations in many-to-one multilingual machine translation.\nWhile pixel representations are not novel, their interaction with multilingual training has not been studied previously.\nText is rendered as an image using a particular font, and then fed into a convolutional network that transforms it into embedding vectors suitable for a transformer network.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "121 Please clarify that Table 4 is found in the appendices."}, "extra_text": {}}, {"review_id": "l4TWuxfp8v", "reviewer": "Reviewer_V6mX", "summary": "", "questions": "QA: Can you please explain the difference of your approach compared to Rust et al, 2023?\n\nQB: Is it possible to check the approach with better multilingual NMT models?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review! We will clarify these points in the final version.\n\n\nQuestion A:  \nOur rendering approach is similar to the one used in Rust et al 2023. The two key differences are a larger patch size of 24px (in order to fit all characters and diacritics from our broader language set), and overlapping strided patches (we use a stride of 12px, while they do not use overlapping patches) which was previously found beneficial for translation (Salesky et al 2021) which we validated on our settings in initial experimentation.  \nExperimentally, the models in Rust et al 2023 were only ever trained on a single language at a time: pretrained on English, and finetuned individually on each target language. The effects of co-training multilingually and whether and why positive transfer or interference may arise with pixel representations were not addressed, nor the data-efficiency of transfer. All experiments in Rust et al 2023 are also classification tasks, not generation. \n\nQuestion B:  \nWhile we agree that it would be beneficial to compare to larger and stronger models, it would not be possible to directly compare to the pretrained models suggested, as it would require training pixel representations on similarly sized corpora, and their training data is both not fully available and beyond our computational resources.¹  \nAdapting pretrained models which were trained with subword embeddings to use pixel representations is a promising future direction but beyond the scope of this work; we will note this in the limitations section of the paper.  \nWe are running experiments on larger corpora than presented here (50M) which we hope to add.  \n\n¹ SMaLL-100 was finetuned from M2M-100 on a randomly selected 456M sentence subset from CCMatrix which was not released. The original M2M-100 data is 16x larger and expensive to extract from the raw releases. The NLLB training data is not fully public.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A novel approach to train NMT model with pixel representation\n2. Better performance compared to traditional subword-style tokenization\n3. The approach also helps better cross-lingual transfer to unseen scripts and is more data efficient", "reasons_to_reject": "1. The technical difference between their approach and Rust et al, 2023 is not clear\n2. Experiment with better multilingual model e.g. NLLB (https://arxiv.org/abs/2207.04672), M2M-100 (https://arxiv.org/abs/2010.11125), and SMaLL100 (https://arxiv.org/abs/2210.11621)?\n", "questions_to_authors": "QA: Can you please explain the difference of your approach compared to Rust et al, 2023?\n\nQB: Is it possible to check the approach with better multilingual NMT models?\n", "paper_topic_and_main_contributions": "In this paper, they proposed a novel approach to train NMT model with pixel representations. They evaluated the model on TED dataset with 7 and 49 languages with a variety of languages and script coverage, and improved the perforamnce compared to previous subword-based studies. Further analysis showed that pixel representation helps better cross-lingual transfer to unseen scripts and is more data efficient.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": " ", "Justification_for_Ethical_Concerns": "please check section 9"}, "extra_text": {}}, {"review_id": "OT1Ki3DHJg", "reviewer": "Reviewer_mFpr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review! We will clarify these points in the final version.\n\nReasons to Reject 1:  \nPrevious work (Salesky et al. 2021, Rust et al. 2023) did this experimentation extensively, so we adapt their best settings. We will clarify why we chose these settings and what experimentation was done in past work in Section 2.1. \n\nReasons to Reject 2:  \nWe will clarify the text here. While we do not ablate whether “ja” improves with “zh” only, we do investigate which of the factors mentioned on L244-250 is most critical to positive transfer (script similarities, language similarities, or simply having additional data) for TED-59 where we have more languages to study (Section 4.1), and find that the amount of data for a given script is most strongly correlated with performance improvements.  \n\nPresentation 1:  \nWe will change this to the mean chrF improvement in the abstract and introduction.  \nWe show chrF but highlight BLEU as it is the only metric reported in previous work on these corpora. \n\nPresentation 2:  \nThank you for this pointer; we have the information necessary to add similar figures to our appendix. \n\nPresentation 3:  \nThe specific languages in Table 2 were chosen because they are the only languages previous work reported performance for individually in addition to aggregated (average) scores. We note this on L264-267, but will ensure this is clearer in the final version and highlight that this set was originally chosen to include related languages with different scripts as well as unrelated languages with similar scripts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "That's a well designed study which shows how multilingual models can avoid the lexical bottleneck for lesser-resourced languages.", "reasons_to_reject": " 1. It is suprising that you haven't experimented with different image conversion parameters, such as different fonts, font sizes and resolutions. Google Noto Sans 10pt at 120 dpi is a very specific setting. The question in Lines 218-220 implies that these variations need to be studied.\n 2. Similarly the statement on the source of improvements as in Lines 244-250 should have been studied. Does performance for \"ja\" improves when you add \"zh\"?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The study proposes a method for solving the problem of the limited lexicon of multiligual language models by replacing subword embeddings with image embeddings for the individual script.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "  1. The claim of \"up to 9 BLEU\" is misleading. Report the mean. And also chrF is more robust for neural models, so it's better to provide your main reporting for it. \n  2. Conneau et al 2020 present a very concise way to describe the contribution of individual properties of their architecture on the overall performance.\n 3. The reasons for choosing the specific languages in Table 2 are not clear. I think the pairing in this table is good as it includes related languages with different scripts  as well as unrelated languages with similar scripts. Please add an explanation.\n"}}]

Author rebuttal:

Meta-review: This paper presents an empirical study on the effect of pixel-level text representations on multilingual translation models. Building on best practices established in previous work, which was trained one language at a time, this work shows that multilingual pixel representations can improve data efficiency and cross-lingual transfer to unseen scripts. Reviewers are excited about the approach as an alternative to subword models that enables positive transfer for low resource languages, and they have few concerns that were not adequately addressed by the response.
Decision: Accept-Main