Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Authors: Yi Tay
Vinh Q. Tran
Sebastian Ruder
Jai Gupta
Hyung Won Chung
Dara Bahri
Zhen Qin
Simon Baumgartner
Cong Yu
Donald Metzler
Venue: ICLR 2022 Poster (2022)
Keywords: transformers
NLP
language

Abstract: State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the character level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive character-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of vanilla character-level Transformers by up to  while maintaining quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

Published as a conference paper at ICLR 2022
CHARFORMER: FAST CHARACTER TRANSFORMERS VIA GRADIENT-BASED SUBWORD TOKENIZATION
Yi Tay;, Vinh Q. Tran", Sebastian Ruder Jai Gupta, Hyung Won Chung, Dara Bahri Zhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler Google Research and DeepMind vitayggoogles com, vgt.ranggoogle.co com
ABSTRACT
State-of- thettrr models in natural language processing rely on separate rigid sub- word tokenization algorithms, which limit their generalization ability and adap- tation new settings. In this paper, we propose a new model inductive bias that learns subword tokenization end-to-end as part of the model. To this end, we introduce soft gradient- gradient-based subword tokenization module (GBST) that automat- ically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using block scoring network. We additionally introduce CHARFORMER, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments English GLUE, multilingual, and noisy text datasets, we show that CHARFORMER outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperform- ing subword-bar based models. Additionally, CHIARFORMER is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free free models that are trained completely end-to-end.
INTRODUCTION
Neural networks have achieved tremendous success in natural language processing (NLP) by replacing feature-engineered models with stacks of functions that are learned end-to-end from vast amounts of data (Mikolov et al., 2013) Peters et al.) 20188) Howard and Ruder 2018). The single component
of the traditional NLP pipeline (Manning and Schütze 1999) that has so far resisted gradient-based learning is tokenization, which is commonly applied as pre-processing step. State-of-the pre-trained language models (Devlin et al., 2019) generally rely on data-driven subword- bbsed tokenization algorithms (Schuster and Nakajima| 2012) Sennrich et al. 2016 Wu et al.) 2016 Kudo and Richardson, 2018) while expert-crafted segmentation algorithms are still common for languages without whitespace separation such as Chinese, Thai, and Korean (cf. Lample and Conneau 2019).
This reliance on rigid tokenization methods introduces a bottleneck into current NLP systems that limits their capabilities. Subword segmentation algorithms split tokens into subwords solely based on frequency, without taking into account lexical or semantic similarity. As a result, models are brittle to rare words (Gong al. 2018) and perturbations, both natural and adversarial (Belinkov and Bisk 2018: Pruthi et al. 2019 Sun et al. 2020). In multilingual models, tokens in low-resource languages those languages and deteriorates cross-
split into many subwords, which impacts performance lingual transfer (Hu et al. 2020 Wang et al. (202). Finally, a separate tokenization algorithm leads to mismatch between the pre-training and downstream distribution of words when adapting pre-trained language models to new settings, which requires significant engineering effort to overcome.
The direct application of character-level modelling into pre-trained language models in results in severely increased computational and memory complexity due to an increased sequence length and generally lower performance. To address this problem, we propose gradient- based subword tokeniza- tion (GBST), new method that combines the compositionality of character- levell representations
"Equal Contribution

---

Published as a conference paper at ICLR 2022
with the efficiency of subword tokenization while enabling end-to-end learning. Our method learns latent subword representations from characters using large amounts of unlabeled data. Specifically, GBST learns a position-wiss soft selection over candidate subword blocks by scoring them with scoring network. In contrast to prior tokenization-fres methods (Clark et al. 2021), GBST learns interpretable latent subwords, which enables easy inspection of lexical representations and is more efficient than other byte-based models (Xue et al., [021)) Given that simply applying standard Transformer on a sequence of characters and bytes is computationally prohibitive, GBST paves the way for usable, practical and highly performant character- level models. A high level overview of how the GBST module is applied can be found Figure (A (Appendix).
We furthermore introduce CHARFORMER, Transformer encoder-decone model that uses GBST to operate directly on the byte level. In addition, we experiment with a re-scaled variant of CHAR- FORMER, which allocates additional capacity to the encoder to make up for the lack of discrete subword embeddings.
We evaluate model on range of standard and non-standard English, and multilingual downstream tasks. On English GLUE and long document classification tasks, CHARFORMER outperforms strong par with subword-based models such as BERT (Devlin et al. 2019) and T5 (Raffel et al. 2020). On toxicity detection in social media datasets (Borkan et al.) 2019 Wulezyn et al., 2017), CHARFORMER outperforms byte-level baselines as models, demonstrating robustness to spelling variation and non-standard language. Finally, multilingually pre-trained CHARFORMER performs par or outperforms strong subword-based multilingual baselines on standard cross-lingual datasets.
byte-level baselines and overall achieves performance
well as subword-basee
We additionally demonstrate CHARROORER is more efficient compared byte-level and subword- based models with similar numbers of parameters. On a comparable setup, CHARFORMER out- performs a baseline similar to the recent state-of-the-arr byte-level model ByT5 (Xue et al. 2021) while being x more memory efficient and 10-93% faster. CHARFORMER also trains 28% faster than the subword-level mT5 model (Xue et al., 2220)), has 3x fewer parameters and achieves comparable quality on well-establisher benchmarks. Finally, we demonstrate via visualization that the latent subwords learned by CHARFORMER are interpretable some extent.
2
CHARFORMER
This section introduces our efficient character-level architecture, CHARFORMER, CHARFORMER comprised of a Gradient-Bass Based Subword Tokenization (GBST) module, followed by deep Trans- former layers. The input to the GBST module is sequence of characters or bytes[] which is then downsampled to construct latent subwords.
2.1 GRADIENT-BAL BASED SUBWORD TOKENIZATION (GBST)
The input to GBST is a tensor of shape E R Lxd where L is the number of input characters and d is the character embedding dimension. The key idea behind GBST is for the model to learn to perform a latent subword segmentation of the input by selecting the most suitable subword block at every character position. A block is a contiguous span of characters Xgi+s of length b for í < L b.
2.1.1 CONSTRUCTING CANDIDATE LATENT SUBWORD BLOCKS
We first enumerate all possible subword blocks of size b up a maximum block size M. In order to learn subword block embeddings, we use a non-parameterized strided pooling function F Rx + d that projects a subword block consisting of sequence of character embeddings X E b to single subword block representation Xb,5 E Rd for block size b at position i. We compute subword blocks Xb,i with a stride s:
X, [F(X+++)); (+)))()()()))
We choose bytes rather than characters (Unicode code points) as this allows us to use vocabulary of 256 possible byte values for all settings. We note that for languages with a Latin alphabet, many characters correspond to single hyte. For other languages, each character corresponds 2-3 bytes in general. For simplicity and align with prior work, will generally talk about characters unless stated otherwise.
2

---

Published as a conference paper at ICLR 2022
h
m
1-Blocks
X
X
%
Blocks
Pee
3-0 Bllocks
X
X
X
X.
Pa Pst Pr.
Per
Blocks
X
%
x K
%
Pv
Pex
(a) Formation of subword blocks to be scored by FR.. Offisets and/or pre-GBST convolutions not shown.
(b) Block scores that have been expanded back length L. Softmax is taken over block scores each position i to form block weights for constructing latent subword representations.
Figure 1: Illustration of subword block formation and scoring.
In practice we set s = b, thus X E R The construction latent subword blocks creates a shorter overall sequence length by downsampling. We construct X₁, for b I,..., M, which can be seen in Figure for M 4.
Considering Offsets A limitation of strided implementation is that it is unable model all possible subword windows. For instance, for the character sequence [a, b, c, d] we would only be able to allocate [a, 6] and [c, d] as subword blocks of length b 2 and would ignore the subword block [b, c). Offsets can be used to model sliding windows of all possible subword blocks. We consider enumerating all possible strided blocks by additionally shifting sequences up until the offset s. this increases computation, we instead propose to first apply ID convolution to X, prior to enumerating subword blocks. This effectively "smoothes" over the subword blocks. We use the variant with ID convolutions in our main experiments and provide additional ablations in $8.3 of the Appendix.
Considering Intra-block Positions It is important to preserve the ordering of the characters within the block X,,11 Xi+b. E.g.. the output of F should differ for the blocks abc and bea. For certain choices of F may be valuable to add a positional embedding (Vaswani et al.) 2017) to Xgi+b before applying F. Note that this positional embedding would only be for individual blocks, and is not global to the entire input sequence. That is, only positional embedding values for positions 1,..., b would be used. However, in practice we apply ID convolution before the GBST layer and use the mean-pooling function for F. We find this to be sufficient to distinguish between same sized blocks with different character orders.
2.1.2 BLOCK SCORING NETWORK
In order to allow the model learn which block select for every character position, introduce a block scoring network. The block scoring network simply a parameterized function FR(() that produces score for each candidate block. Given subword candidate block X,,5 Rd, we compute score pb, associated with the block using simple linear transformation FR IRd R:
pb.; = Fr((X..)
We perform ranking of subword blocks with regard to each character position in the original sequence. At every position i, the model learns select the most suitable subword block Xt,, among all block sizes 1 < b S. M. As each sequence of subword blocks X₁ is downsampled, we realign the representations of the subword blocks by upsampling each X, to its original sequence length Specifically, for a block size of b, we replicate each block representation X',; b times. We then score each candidate block at each position using the softmax function:
P: = softmax ,PM,a)s
(3)
which computes a relative score of each candidate block at each position and P. e RM We show the scoring of realigned blocks in Figure
3

---

Published as a conference paper at ICLR 2022
2.1.3 FORMING LATENT SUBWORDS
We then sum the representations all subword blocks Xt,5 at each position multiplied by their learned probability Po.s to form latent subword representation X; E R².
M
X,
Σ Xb.i
(4)
Intuitively, the model learns an ideal subword block for each position. In contrast to standard deterministic subword tokenization algorithms, this selection is soft and can thus consider different possible segmentations every position i. In general, however, this formulation still assumes that subwords are contiguous sequences of characters. While additional context can be considered via the convolutions in $2.1.1] non-concatenative morphology where morphemes are discontinuous may be harder for the method to model
2.1.4 POSITION-WISE SCORE CALIBRATION
In the above approach, the scoring of each position independent of other positions. We hypothesize that it may be beneficial for block scores at each position to be aware of each other. To this end, we introduce an optional module that enables learning a consensus among block scores by calculating dot products across the scores P. across all positions i L]. This be viewed as form of self-attention across block scores, albeit without any projections for computational efficiency. To learn the new scores P E LRLMM we compute P = soft (PT))
2.1.5 DOWNSAMPLING
After learning candidate block or mixture of blocks for each position, we use downsampling func- tion FD : 4xd + R d xd that downsamples the sequence of latent subwords X = [XI, Xr] to X, reducing its sequence length by a factor of d,. We choose Fn to be non parameterized mean pooling operation. Notably, such simple stride-based pooling removes potential redundancies caused by adjacent positions selecting similar bloeks as the mean pool of two identical block embeddings produces the same outcome. Intuitively, as the downsampling operation is fixed, the parameterized components preceding it should learn an optimal subword tokenization given the downsampling.
2.2 TRANSFORMER STACK
The remainder of the CHARFORMER model remains identical to a regular Transformer encoder- decoder model. The Transformer stack operates on the downsampled latent subwords instead of subword embeddings.
Re-scaling of the Transformer Stack While subword- models allocate much their capacity to subword embeddings up to 71% of all parameters for contemporary multilingual models (Chung et al. (02T)-, the character vocabulary of character-level models much smaller and thus less expressive. Similar Xue et al. (2021), 3, we hypothesize that character-level models require deeper encoder stacks than subword-based models to make up for their smaller embedding capacity. Conse- quently, we explore scaling variant of CHARFORMER that puts more parameters the encoder at the expense of the decoder while preferring deep narrow model over larger wide model. Specifically, we re-configure the Base model size to be similar the T5 Small model size, with an expanded (Scaled Base) has 134M parameters,
24 layers in the encoder. The resulting CHARFORMER which is about 67% the parameter footprint of the standard base T5 model (200M parameters: |affel et al. 2020). Moreover, this particular CHARRFRRMER model is approximately 50- (0-100% faster than T5 base model (see $ For the re-scaled variant, we also used the GLU variant described in (Shazeer 2020) which is commonly referred to the V1.1 variant in the T5 library.
Future work could explicitly seck model discontinuous morphological processes by considering skip- grams addition to character n-grams, although this would increase computational costs. 'The henefits of such re-scaling have also heen observed for subword-based encoder-decoder neural machine
translation models (Devlin 2017 Kasai
2021.
4

---

Published as a conference paper at ICLR 2022
A Note on Comparing Character-level and Subword-based Methods Prior work on efficient methods generally compares models with the same number of parameters (Chung et al.) (021). However, whereas embedding look-up even with large vocabularies in subword-basd
methods is
0(1), redditribbttngg the subword embedding parameters in character-level models such as ByT5 (Xue al., 202T) dense layers incurs much higher computational costs-a a 25% penalty in training speed. We believe that a fair re-scaling of character-level models should not only aim to match the number of parameters but also the compute and inference costs of subword-based models under the assumption that char/byte-level models will require longer sequences (see $ for comparison).
Span-based Pre-training Our pre-training scheme follows T5 quite closely. We mask N contiguous characters and train to predict them in a -oseeueee--- architecture following Xue et al. (202I). The model optimizes the cross-entropy loss and is trained with teacher forcing.
EXPERIMENTS
We evaluate our method both in English as well as in multilingual setting relevant benchmarks and compare against state-of-the-ar character-level and subword-bass, methods.
3.1 EXPERIMENTS ON MONOLINGUAL ENGLISH DATASETS
Data To showcase the effectiveness of the proposed method, we evaluate on a diverse set of standard English tasks from GLUE covering sentiment classification (SST-2: (Socher al. 2013), natural language inference (MNLI, QNLI; Williams et al. 20188] Rajpurkar et al. 2016), paraphrase detection (Dolan and Brocketi 2005 MRPC, QQP) and sentence similarity (Cer et al.) (017). In addition, we evaluate on tasks that require dealing with long documents, both for sentiment analysis (IMDb; Maas
al. 2011 and news classification (AGNews: Zhang et al., 2013).
Baselines We compare CHARFORMER against the following state-of-the-ane habword-based models: BERT (Devlin al., 2019). an encoder-onj pre-trained masked language model; and T5 (Raffel
et al. 2020), encoder-decoder model. We also compare against Byte-level T5 (Xue et al. 202 )), T5 model that is directly applied to bytes. We additionally evaluate the impact of the downsampling in CHARFORMER by comparing it to the downsampling used by the character-level CANINE (Clark ctt al., (02T) model in our framework. CANINE downsamples a character sequence using local attention and pooling via strided convolutions. As the original CANINE uses encoder-only model and was only trained on multilingual data, we integrate CANINE-style downsampling into Byte-level T5, which we refer as Byte-level T5+LASC (local attention-strided convolution) As an ablation for the GBST inductive bias, we compare against Byte-level T5+Conv Base a convolutional baseline of Byte-level T5 with ID convolution of filter size placed before the encoder. Note that in all the baselines and for CHARFORMER base models, in the spirit of fair comparison, we compare them at an equal parameterization (size). Our scaling experiments are reserved for our S Sass models, which is intended to only be compared with subword T5 models, and not to unscaled byte-level baselines. Finally, we include an S Base scaled version of Byte-level T5 for comparison.
Setup We evaluate Base and SBase configurations of CHARFORMER with 203M and 134M parameters respectively. We compare to Base configurations of BERT and T5 that have a similar number of parameters. We pre-train all models on the C4 corpus for IM steps using batch size of 64 and sequence length of 1024. All non-subword models use a vocabulary of 256 bytes Our pre-training scheme corrupts spans with mean length of 20 bytes. Each model is pre- treitrannd on 16 TPU V3 chips. We pre-train our models with the Adafactor optimizer with an inverse square root learning rate. We then fine-tune on each individual task separately using constant learning rate of 10-3, More details can be found in the Appendix.
'Compared CANINE, Byte-level LS+LASC does operate Unicode codepoints and has decoder. It thus forgnes character hash emheddings and upsampling pracedures respectively. Following Xue al. 2021 we discard illegal UTF-8 sequences and reuse the final 100 byte IDs sentinel
tokens.
5

---

Published as a conference paper at ICLR 2022
Table 1: Comparison of CHARFORMER against other subword and character-level models with
different parameter sizes on diverse standard English datasets.
Model
[0]
SST-2
MNLI
QNLI
MRPC
QQP
STSB
COLA
AVG
BERT Tosce,Sburore
1 10M
92.7
84.4/-
88.4
86.7/-
T5 Subword
220M
92.7
84.2/84.6
90.5
88.9/92.1
91.6/88.7
88.0
53.8
84.3
Byte-level T3 Base
200M
91.6
82,5/82.7
88.7
87.3/91.0
90.9/87.7
84.3
45.1
81.5
Byte-level TS+Conv
205M
89.8
81.1/82.5
89.2
83.6/89.2
90.7/87.7
85.0
47.1
81.2
Byle-level TS+LASC
205M
90.0
80.0/80.8
87.1
82. 8/88./8.1
89.0/85.4
83.7
25.3
77.0
CHARFORMER
203M
91.6
82.6/82.7
89.0
87.3/91.1
21.2/88.1
85.3
42.6
81.4
Byte-level Dase
133M
91.2
83,9/83.7
90.9
85.5/89.22
1/.1/88.1
85.7
49.3
82.6
CHARFORMERgPOw
134M
91.5
83.7/84.4
91.0
87.5/9144
91.4/88.5
87.3
51.8
83.6
Table 2: Results on comment classification on Civil Com-
Table Results on text classification on long documents. Model
ments and Wiki Comments. Metries are accuracy and AUC-PR. T5 baseline results are from (Tay et al. (021).
IMDb
News
Model
Civil Comments
Wiki Comments
T5 Subuord
81.2/
91.5/-
Re Subward
94.2
Byte-level T5 Base
91.5
93.6
Byte-level T5
82.8 78.7
93.2 75.4
Byte-level T5+LASC Bace
91.1
93.5
Byte-level 15+LASC
82.9 / 78.2
93.0/75.0
CHARFORMEK gase
91.5
94.0
CHARFORMER Rase
83.0 78.8
92.7 79.77
CHARFORMER
94.4
94.1
CHARFORMER,S.o.
83.0/ 78.9
93.5/75.5
Results For all result tables, we divide the table into three sections: subword baseline(s), un-scaled byte-level baselines, and scaled CHARFORMER results. If section and task combination has more than model result, underline the best result. We show result for GLUE in Table CHAR- FORMER outperforms other character-level baselines trained under the same conditions with the same number of parameters across all tasks, while being considerably faster and requiring less compute than Tt-style models that are directly applied to bytes or characters (see $4). CHARFORMER 5 Base performs even better despite having smaller number of parameters compared to the Base configu- ration, demonstrating the usefulness of rescaling the transformer stack for character-level models. CHARFOR MER M Base furthermore is the only model that performs par standard subword-basse models on some tasks in standard English. In Table 3 we provide results for classification of long documents. Here, CHARFORMER Base is the only byte-level model Base
outperforms the
to outperform T5 Base, on the IMDb classification task, and both CHARFORMER models outperform byte and subword level baselines on AGNews.
3.2 EXPERIMENTS ON NON-STANDARD ENGLISH DATASETS
The previous set of experiments demonstrated the ability of CHARFORMER to perform well on clean datasets consisting of standard English. However, character- level models are particularly suited to data that is noisy, containing spelling variations, typos, and other non-standard language.
Data To demonstrate CHARFORMER'S ability perform well on such data, we evaluate toxicity detection using the Civil Comments (Borkan et al. (2019) and the Wikipedia Comments (Wulczyn al., 2017) datasets. Both are standard benchmarks that require estimating the toxicity of user- generated content. We use the same setup as for the standard English datasets.
Results show results in Table Character-level models outperform the subword-based model on both datasets, demonstrating their suitability to deal with such noisy, user-generated data. CHARFORMER achieves performs on par or outperforms other character-level methods on both datasets across the different model sizes.
3.3
MULTILINGUAL EXPERIMENTS
Data To evaluate the effectiveness of character-level models on multilingual data, we evaluate on standard cross-lingual question answering and classification tasks. In particular, we evaluate on the question answering tasks MyDiQA-Goll -GoldP (Clark et al.) 2020), XQuAD (Artetxe et al. 2020). and
MLQA (Lewis et al. 2020) well the natural language inference task XNLI (Conneau et al. 2018) and the paraphrase detection task PAWS-X X (Yang et al. 2019) from XTREME (Hu et al. 2020). We
evaluate the in-language multi-task setting for TyDiQA-GoldP (Clark al. 2020) where models

---

Published as a conference paper at ICLR 2022
Table 4: Multilingual comparison of CHARFORMER against subword and byte-level models on in-language multi-task, translate-train multi-task, and cross-lingual zero-shot (training on English) settings. Model sizes are the same those in Table (Xue et al. 2020).
mBERT and mT5 baseline results are from
In In-Lanuaage
Translate- Train-All
Sero-Shot
Model
[0]
TyDiQA-GoldP
XQuAD
MLQA
XNLI
PAWS-X
XNLI
PAWS-X
mBERTBaa Bosr (Subword)
179M
77.6/68.0
++-
-f-
65.4
81.9
miae (Subword)
582M
80.8/70.0
75.3/59.7
67.6/48.5
75.9
89.3
25.4
86.4
Byte-level Base
200M
75.6/65.4
68.6/54.3
61. 61.8/44.
69.4
87.1
57.4
80.9
Byte level TSeLASC
205M
70.6/59.7
66.8/52.1
58.8/4111
67.9
84.8
55.2
79.0
CHARFORMER Busc
203M
75.9/65.6
70.2/55.9
62.6/44.9
71.1
87.2
57.6
81.6
CHARFORMER SER
134M
79. 19.1/68.8
73.6/59.0
66.3/48.5
72.2
88.2
66.6
85.2
CHARFORMER CHARFORMER Bere.
134M
81.2/71.3
74.2/59.8
67.2/49.
72.8
88.6
67.8
83.7
Table 5: Comparison of pre-training compute metries for mT55 (Subword) versus comparable qual- ity CHARFORMER models on the mC4 dataset. 64 TPUv3 chips were used for this experiment. CHARFFORER sees the same number of tokens after downsampling as mT5 Bases while CHARFOR Long. roughly sees the same amount of raw text as m'T5 given that SentencePiece subword token is about 4.1 bytes on average (Xue et al.) 2021). CHARFORMER
is 28% faster than mT5 Bese, while using 33% of the FLOPS.
Model
Batch Size
L
d,
[0]
Speed (steps/s)
FLOPS
mTSs (Subword)
1024
1024
582M
1.54
1.3 x 10
CHARFORMER
1024
2048
2
134M
1.98
4.3 x 10t4
CHARFORMER CHARFORMER LonPT
2048
2048
2
134M
1.01
4.3 x 10
are fine-tuned on the combined gold data in all target languages and the translate- train-all setting where models are fine-tuned on English training data plus translations in all target languages for the other datasets. Both are the best-performin settings for the respective tasks in (Hu et al. 2020). In addition, we evaluate on zero-shot cross- cross-lingual transfer from English on XNLI and PAWS-X.
Baselines
We compare to strong multilingual subword-ben bbsed baselines including multilingual BERT 2019) and multilingual T5 (Xue et al. 2020). In addition, we compare to the byte-level $3.1 which we pre-train on multilingual data.
(Devlin et al.
models from
Setup We pre-train CHARFORMER as well as the Byte-level T5 and Byte-level T5+LASC baselines on multilingual mC4 Common Crawl (Xue et al. 2020) in 101 languages. Base size models were trained for 1M steps using a batch size of 64 and sequence length of 2048, with the exception of Byte-level T5 Bases which was trained with sequence length of 1024, as training speed was prohibitively slow (see Table IT). CHARFORMER and CHARFORMER Long pre-training) are trained with larger batch sizes for fair comparison with mT5. In particular, CIIAR- FORMER pre-trains on the same amount of tokens after downsampling as mT5 Bases while CHARFORMER Bose, LongPT pre-trains on roughly the same amount of raw text mT5 Base, given that a SentencePiece subword token is about 4.1 bytes on average (Xue et al.) 202T); ; see Table 5 for further details. All models were fine-tuned with input sequence length of 4096 for question- answering tasks and 2048 for inference tasks. Score calibration was not used for these experiments, as it did not benefit the model in the multilingual setting. For XNLI and PAWS-X (both translate-train and zero-shot settings), we also observed that performance improved if the GBST layer was not updated during fine-tuning; the reported CHARRFRMER numbers reflect this configuration. Otherwise, all other soper-parameter and model sizes are unchanged from the English experimental setup.
Long (longer
Results We show in-language multi-task, translate- train, and cross-lingual zero-shot results in Table CHARFORMER Base is competitive with standard subword-based models and CHAR- FORMER PoT outperforms subword-based models on TyDiQA-GoldP (in-language multi- task). Additionally, in the translate-train setting CHARFORMER (Bngse,.oog.P is on par with subword models on XQuAD and MLQA, and close to parity on PAWS-X. Furthermore, CHARRORMER outperforms other character-level models in the zero-shot setting. However, we observe that this setting still remains a challenge for token-free models in general. We hypothesize that model size may be major factor here. Finally, we provide additional comparison between GBST and LASC at a fixed down-sampling rate in Section 8.4 (Appendix), showing that GBST significantly outperforms I LASC on TyDiQA
7

---

Published as a conference paper at ICLR 2022
Table 6: Pre-training compute metrics of models at different input lengths, downsampling rates, and model sizes on the English C4 dataset. 16 TPUv3 chips were used for this experiment, These numbers reflect batch size of 64. Memory refers to per-device peak memory usage on TPUv3 chips.
Model
d.
[0]
Speed (steps/s)
FLOPS
Peak Mem.
T5 Buse (Subword)
512
220M
9.3
1.1 X 10
Byte-level T5 Basc
1024
200M
8.2
2.9 X 10
3.09GB
Byte-level Bas
1024
4
205M
15
9.9
10
1,62GB
CHARFORMER Base
1024
206M
11
1.6 10
1.9GGB
CHARFORMER
1024
203M
15
1.1
10th
1.63GB
CHARFORMER Base Base
1024
134M
1.3
10th
1.73GB
CHARFORMER
Base Base
1024
134M
20
8.7
10
1.34GB
01
C3
33
04
62
G4
A3
54
61
65
0
33
93
50
XX
01
n4
0.
87
x1
6.
Charactors
Figure 2: Visualization of block scores (softmax weights) for every byte position from multilingual CHARFORMER on an example English input.
4 SPEED, MEMORY AND PARAMETERS
Table 6 reports the speed (global training steps per second), parameter sizes and number of float- ing point operations (FLOPS) for each forward pass of the models used in our experiments. All experiments were
16 TPU-v3 chips and speed is benehmarked on English C4 pre-training at the IK input length (L.). CHARFORMER models are generally more efficient both in terms of speed and FLOPS compared to other character-level models at different parameter sizes. With low down-sampling rate d, for CHARRFRRER,, Byte-level TS+LASC is more efficient due to using higher down-sampling rate. Directly consuming the character sequence with Transformer model is slow and requires a large number of FLOPS, which is exacerbated with longer sequence lengths where Byte-level T5 is more than 2x slower than the fastest CHARFORMER. This difference even larger at longer input sequence lengths, which we report in the Appendix. CHARFORMER Base achieves better performance (see } with fewer parameters but more FLOPS by using a deep thin encoder and is twice as fast as the subword- based model with similar performance, T5 T5Base-
5 VISUALIZING LATENT SUBWORDS
One benefit of CHARFORMER compared to other character-level methods is that the subwords it learns are directly interpretable and may give some indications to the behaviour of the underlying model. We visualize the scores the multilingual CHARFORMER has learned to assign to subword blocks of different sizes for the string 'on subword tokenization' in Figure We observe that the model learns to allocate single-character subword blocks predominantly to vowels and whitespace in English. Moreover, in English the model allocates larger subword blocks to the beginning and end consonants of subword. Together, we believe this suggests that the model has learned meaningful segmentation of the input, and that is able to dynamically mix between byte-level and subword-leve features. Such behaviour could also parallel the relative importance attributed consonants for word identification observed during reading in humans (Lee et al. 2001
Carreiras et al. 2008).
6 RELATED WORK
Subword tokenization Standard algorithms for deterministic subword tokenization are Byte Pair Encoding (BPE: Sennrich et al.) 2016), Wordpiece (Wu et al. 2016)), and SentencePiece (Kudo and Richardson 2018). Prior work has highlighted issues with some of these algorithms (Bostrom and Durrett 2020) and has generally observed that models learned with such rigid tokenization do not cope well with variation in language (Sun et al., 2020). To make a model more robust to morphological and compositional generalization, probabilistic segmentation algorithms such as
8

---

Published as a conference paper at ICLR 2022
subword regularization (Kudo 2018) and BPE- (Provilkov et al. 2020) have been proposed, which sample different segmentations during training. Recent methods propose to make models more robust for downstream tasks by enforeing prediction consistency between deterministic and probabilistic segmentations (Wang et al. 2 and propose to update the tokenizer based on the downstream loss under different segmentations (Hiraoka al., 2020 2021). [He et al.] (2020) proposed DPE (dynamic programming encoding), a segmentation-ben based tokenization algorithm based on dynamic programming. Such methods, however, incur large computation costs due multiple forward passes needing to be performed for each segmentation of an example or due to the expensive DP computation, which make them unsuitable for pre-training.
Character-level models For recurrent neural networks, pure character-level models that take a sequence of characters as input (Gravess 2013 Zhang et al. 2015 Hwang and Sung 2017) have mostly been superseded by character
methods that compute token-level representation using CNN over characters (Kim et al. 2016 Jozefowicz al.) 2016 Peters et al. 2018) due to poor performance when learning directly from characters. Such character-aware representations have lately been applied to deep Transformer models (EI Boukkouri et al. 2020 Ma et al. 2020). These methods, however, still require tokenization for prc-processing and cannot be directly applied to languages without whitespace separation. Prior work also learned segmentation as part of the model but did not scale very well (Wang al. [2017] Kreutzer and Sokolov 2018 Kawakami et al. 2019). One notable exception is (Lee et al. 2017). which enabled fully character-level neural machine translation, using stacked convolutions, max pooling, and highway networks. Building on this, recent tokenization-free approaches such as CANINE (Clark et al. 2021) revisit the original character-level setting the context of large pre-trained language models with focus on multilingual models. Our method outperforms CANINE-style downsampling (local attention, strided convolutions) and also leads to improvements in the monolingual setting, while using less compute and parameters to down-sample than both Lee et al. (0117) and Clark et al. (202I). Recently, ByT5 (Xue et al. 2021T) set new start-of-the-at results for tokenization- models, by operating on the byte-level. This work performs par with outperforms ByT5, with significant gains in speed and compute efficiency.
Multilingual models Current multilingual models are generally analogues to successful monolingual Transformer models (Ruder et al., 2021). Consequently, models such as multilingual BERT (Devlin et al. 2019) and XLM-R (Conneau et al. 2020) employ the same subword tokenization algorithms as monulingual models, now applied to massively multilingual corpus. In the multilingual setting, the problems of subword over-segmented while figh-frequency tokens are under-segmented, which limits cross -ongull transfer (Wang et al. 202T). This motivates our work as well as recent work on character-leve models.
tokenization are exacerbated as tokens in languages with few data are
Efficient Transformers Moving from subwords characters significantly increases the sequence length, which is an issue for Transformers due to the quadratic complexity of self-attention. Many ef- ficient self-attention models have been proposed (Choromanski et al. 2020 Wang al. 2020; Zaheer et al. 2020 to tackle this problem; see (Tay et al. a) for a comprehensive overview. Notably, the CANINE model uses local attention (Parmar al.] 2018), which could also be swapped with another efficient Transformer variant. We note that the problem of efficiency is important but not the only challenge towards developing performant tokenization-fret models. While applying an efficient attention mechanism might solve the fundamental computational costs of employing character-level models, there is no guarantee that these models will learn locally meaningful compositions.
7 CONCLUSION
We have proposed CHARFORMER, a re-scaled Transformer architecture that integrates gradient-basee subword tokenization, novel lightweight tokenization method that enables efficient end-to-end learning of latent subwords directly from characters. We have demonstrated that English and multilingual variants of CHARFORMER outperform strong character-level baselines across various datasets while being more efficient. CHARFORMER achieves performance on par with subword-base models standard English tasks and outperforms subword-based models on noisy social media data. On multilingual data, CHARFORMER generally performs on par with subword-based models, while being faster than both byte-level and subword-level baselines. Finally, we provide a method to inspect the inner workings of the GBST module. Overall, we believe that the strong results presented in this paper pave the way for highly effective and powerful token-free free models.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We would like to thank Jon Clark, Noah Constant, and Kris Cao for valuable feedback on drafts of this manuscript.
ETHICS STATEMENT
Standard subword tokenization algorithms produce segmentations that do equally represents words and phrases in different languages. Instead, they are biased towards languages that already have many resources available, which leads multilingual models performing worse on under- represented languages (Wang et al., 2021). Tokenization- free approaches such as the one proposed in this paper may help ameliorate this to some extent. Another challenge using large multilingual models in practice is their relative computational inefficiency, which makes them unsuitable in resource-constrained settings common scenarios where under-representee languages are spoken. CHARFORMER trains 28% faster than mT5 and has 3x fewer parameters, may be more suitable choice in such settings compared to state-of- state-of-the-ar multilingual models.
REPRODUCIBILITY STATEMENT
All code train the core byte-level Transformer encoder-decoder for CHARFORMER its variants is already open-sourced as part of the Mesh Tensorflow Shazeer et al. 2018), T Raffel et al. 2020), and ByTs (ue al. 2021 libraries. Additionally, an implementation of Charformer GBST compatible with existing open- source models has been open-sourced" d Tensorflow implementation of GBST in Section 8.77 of the Appendix. All detailed experiment and hyperparameter settings required to reproduce our experiments can be found in Section 8.2 of the Appendix.
We also include a simplified
REFERENCES
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020, 2020. URL
abs/1910. 11856
Yonatan Belinkov and Yonatan Bisk. Synthetic and Natural Noise Both Break Neural Machine Translation. In Proceedings of ICLR 2018, 2018. URL http::/arxiv.....//////// http:://aaxwww/r//////// wwtp:/s/g//aaaaaaa.aa.aaa aas///bs//11
02173
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. CoRR, abs/1903.04561, 2019. URL URL (ttt:://aaxxiiii
org/ abs 1903. 04561
Kaj Bostrom and Greg Durrett. Byte Pair Encoding is Suboptimal for Language Model Pretraining. In Findings of EMNLP 2020, pages 4617-4624, 2020. doi: 18.18653/v 1/2020. 1ttp:3///1/222..niiiii.. wtw.:/...iiniiignemn.......
Manuel Carreiras, Margaret Gillon-Dowen Marta Vergara, and Manuel Perea. Are vowels and consonants processed differently? event-related potential evidence with delayed letter paradigm. Journal of Cognitive Neuroscience, 21(2):275-288, 2008.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarity-multiiinn and cross-lingual focused evaluation. arXiv preprint 1rXiv:170800005.1 2017.
fttrass//tthuuo.cc///ooorrrr///
charformer
10

---

Published as a conference paper at ICLR 2022
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14733 2020.
Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. Rethinking Embedding Coupling in Pre-trained Language Models. In Proceedings of ICLR 2021, 2021.
Jon Clark, Tom Kwiatkowski, Jennimaria Palomaki, Michael Collins, and Dan Garrette. TyDi QA: A Benchmark for iiformatiionSeekingg Question Answering in Typologically Diverse Languages. In Transactions of the ACL, 2020.
Jonathan H Clark, Dan Garrette, Iulia Ture, and John Wieting. Canine: Pre-training efficient tokenization-Fii encoder for language representation. arXiv preprint arXiv:2103.06874, 2021.
Alexis Conneau, Guillaume Lample. Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating Cross-lingual Sentence Representations. In Proceedings of EMNLP 2018, 2018. http://://aaaxi wwt.oogg.//.o...ooooooo.oo.....
htt//bbs/1000 1809. 05053
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: acl main. 747.
URL 122.33//1/02.aalllc/
httpss// //www acl web. org/ www.:wwwww.woooooooogoeeoooo
Jacob Devlin. Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the epu. arXiv preprint arXiv: /7050/9905.//91, 2017.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL 2019, 2019.
URL httt:: org/ wwwpg//as//111.. 04805
William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases.
In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun' ichi Tsuji. CharacterBERT:: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters. In Proceedings of the 28th International Conference on Com- putational Linguisties, pages 6903-6915, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10.11865//1/2020.coo
main.609. URL
https://wwit acl web..we.b..... org/ antho htw.hhhoooggoo.....0oo
0020 col ing -minnngmmaiiin... main. 609
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tin-Yan Liu. FRAGE: Frequency- Agnostic Word Representation. Proceedings of NIPS 2018, 2018.
Alex Graves, Generating sequences with recurrent neural networks. arXiv preprint arXiv: 1308::100..0800 0850, 2013.
Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. Dynamic programming encoding for subword segmentation in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 3042-3051, Online, July 2020. Association for Computational Linguistics. doi:
URL https:/ //ww: /www
ac web. www.oogg/aaaooo antho ogy/2020.00.....ooooo acc ma in. 275
Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. Optimiz- ing word segmentation for downstream task. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 1341-1351, Online, November 2020. Association
for Computational Linguistics. doi: 10. 16.18653/v1/2020 http://000..nnninneemm....p..pp. 120. URI. https
7/www ac web. www.oorgatttto hnttoo cgy 2020. . f ndi . 120
Tatsuya Hiraoka, Sho Takase, Kei Uchiumi, Atsushi Keyaki, and Naoaki Okazaki. Joint Optimization of Tokenization and Downstream Model. In Findings of ACL-IJCNLP 2021, 2021. http
7/ 7/arxiv. wttp:i.wrrr/bbb//.0555. 12410
11

---

Published as a conference paper at ICLR 2022
Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018, 2018. URL [tRl::t////ax...oo httgg//bs/180..... 00101.01155
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual General- ization. In Proceedings of ICML 2020, 2020.
Kyuyeon Hwang and Wonyong Sung. Character-level language modeling with hierarchical recurrent neural networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5720-5724. IEEE, 2017.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arviv preprint arXiv: 10X.v:1600.022100 2016.
Jungo Kasai, Nikolaos Pappas, Hao Peng. James Cross, and Noah A. Smith. Deep Encoder, Shallow Decoder: Reevaluating con-autoreeressive Machine Translation. In Proceedings of ICLR 2021, 2021. ISBN 0080437516.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. Learning to discover, ground and use words with segmental neural language models. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 6449, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1645. URL htthh:p////// www. www.wb.clwbb... org/
Onthology/P19-110
Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush. Character-aware neural language models. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Julia Kreutzer and Artem Sokolov. Learning to segment inputs for nmt favors character-level processing, 2018.
Taku Kudo. Subword regularization: Improving neural network translation models with mul- tiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/11/118-1007 URL
ht https: / www. wwwwwb.wwb...... org/ anthology/PP8-220
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, November 2018, Association for Computational Linguistics. doi: 10.18653/v1/ D18-2015 W.RLbttps::/// www.ww/wwwww...... org. antha htt.:///r....attooooo//eBeee. 2012
Guillaume Lample and Alexis Conneau. Cross-linguall Langlliii Model Pretraining. In Proceedings of NeurlPS 2019, 2019. URL httpps://gi http:::///iituuu.ooo googl e reeeerrrreeebbrrrreeeee
Hye- Won Lee, Keith Rayner, and Alexander Pollatsek. The relative contribution of consonants and vowels word identification during reading. Journal of Memory and Language, 44(2):189-20) 2001.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. Fully character-level neural machine transla- tion without explicit segmentation. Transactions of the Association for Computational Linguis- tics, 5:365-378, 2017. doi: 10.1 a 00067. URL htLhstts:: http://aaaannntooo.ooooooo// Q17-1026
Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating Cross-lingual Extractive Question Answering. Proceedings of ACL 2020, 2020. URL http:
//arxiv. wwtp:////...bbbbb.1...... 07475
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. CharBERT: Character- aware prained language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 39-50, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 11211663/1120222coliggg main.4. (tLhttp
7/ 7/www. ac www. wwb.org/anthoo...ooooooooooo wew.oorg///ttthooooo////.......
12

---

Published as a conference paper at ICLR 2022
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
Christopher Manning and Hinrich Schütze. Foundations of statistical natural language processing. MIT press, 1999.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, 2013.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, In Proceedings of NAACL-HLT 2018, 2018
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1882-1892, Online, July 2020. Association for Computational Linguisties. doi:
10. 12.18653/v1/2020.00 mal-main. 170. URL https: www. awwweb..lww.oo.. org wttp/annhooooy//2222222
acl main. 170
Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1582-559,, Florence, Italy, July 2019. Association for Computational Linguistics, doi: 10. 10.18653/v1/P19 1561. URLhttts:: www wwwwwbw............. org/
aantholoyye19-15611
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Test Text Transformer. Journal of Machine Learning Research, 21, 2020. URL http:
//axxiv.or//abs: 101.9911....... 10683
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language P Processing, pages 2389- Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18.188//111D16. 1264. URL http::ttp////e /www ac web
org/antholoogyDIee 1264
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. Xtreme-r: Towards more challenging and nuanced multilingual evaluation. arXiv preprint arXiv:2104.07412 2021.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152 IEEE, 2012.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/11P16-1100 URL
https: www www.wacwweb.or.... httpbborgg/aaaooooo//e--e-----
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.0520,, 2020.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow; Deep learning for supercomputers. arXiv preprint arXiv: /8/ 1.00844 2018.
13

---

Published as a conference paper at ICLR 2022
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pages 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL htt.hhttps:///tt www.wwawllwwe... www.cllebborgo.... http://wwccc.e...rrr////aooooooa http://aaaaaoooooo111111111
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li. Philip Yu, and Caiming Xiong. Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003. 040,: 2020.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 200020001.00066, 2020a.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint ar XiXx:2009900632, 2020b.
Yi Tay, Mostafa Dehghani, Jai Gupta, Dara Bahri, Vamsi Aribandi, Zhen Qin, and Donald Metzler. Are pre-trained convolutions better than pre-tree preitrained transformers? arXiv preprint 10Xv:2205..0322, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. U. V. Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances
Guyon,
Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: //pposedd////oooeeene/e/s hewpeinnn..nnnnugeeeeeeeee neurips. ccc/paper/2011/F11 3f5ee243547dees fadee331laaa5acccccc.ccc off pd
Chong Wang, Yining Wang. Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. Sequence modeling via segmentations. In International Conference on Machine Learning, pages 3674-3683. PMLR, 2017.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
Xinyi Wang, Sebastian Ruder, and Graham Neubig. Multi-view Subword Regularization. In Proceedings of NAACL 2021, 2021. URL http:///xxxvv.o../aaooo
08490
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1//N18-110011 URL https: www.wc.w.ccc.................www www..aawwwb..rggaattll..ooo Mtw.o/ooooyy/-111111111
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint Xivi 08144, 2016.
Ellery Wulezyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW '17, pages 1391- 1399, Republic and Canton of Geneva, CHE, 2017. International World Wide Web Conferences Steering Committce. ISBN 978 9781450349130
doi: 10.114573038912.3355999 URL https?
//doi www.orrg110. 0ttp://0..1155/30331110 S0..333339922...35313
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer, 2020.
14

---

Published as a conference paper at ICLR 2022
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. org/ 2ttps//205..166
arXiv preprint ar arXiv:2105.136200 2021.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification. In Proceedings of EMNLP 2019, 2019. URL http:
11828
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.1 /4062, 2020.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems, pages 649-657, 2015. URL http://arx org/ wwggabbs/109..o.. 626
15

---

Published as a conference paper at ICLR 2022
8 APPENDIX
8.1 OVERVIEW
dudinedd rabring
Updaned
Transformer Stack
Transformer Stack
Subword Token Sequence
Soft "Subword" Sequence
Gradient-baser Subword Tokenizer (GBST)
Subword Tokenizer
Sequence
Sequence
Subword Model
Charformer
Figure 3: High-level differences between traditional subword Transformer models and Charformer which uses gradient-based subword tokenization.
8.2 HYPERPARAMETERS
This section describes the hyperparameters that we use in our experiments.
Monolingual English Datasets Our small model follows the T5 small model size with encoder layers and 6 decoder layers, hidden size dmodet of 512, 8 heads, de. of 32 and djf of 2048. This corresponds to bi_vl vV__smlll_gnn in the T5 codebase. The base model (corresponding to bi vi_yllii)) has 12 encoder layers, 12 decoder layers, dmodei of 768. d of 3072 and 12 heads. The SBase model has 24 encoder layers and 6 decoder layers, while the remainder of its hyperparameters remain identical the small model. All Transformer stacks
relative attention over positional encodings as per (Raffel et al., 2020). For pre-training, we run our models for 1M steps on C4 with a batch size of 64. The maximum sequence length for all tasks is set 1024. TPU packing not activated for Charformer. For Charformer, the filter size of the pre-GBST convolution is set to 5 by default. For CHARFORMER, the downsampling rate is tuned in the range {2, 3, 4). For smaller models, the rate of 2 seems to work consistently the best. For base models, the best models used a downsampling rate of either 2 or 3. For the SBase models, the optimal downsampling rate was often 3.
Multilingual Datasets Hyperparameters are kept constant between English and multilingual tasks except for the following differences. For pre-training, we run our models for 1M steps with batch size of 64, except for CHARFORMER FORMER Po which usees a batch size of 2048. Models were pre-trained with a maximum sequence length of 2048 and fine-tuned with a maximum sequence length of 4096 for TyDiQA XQuAD, and MLQA, and 2048 for XNLI and PAWS- X. Byte-level T5 Base was the only model to be pre-trained with maximum sequence length of 1024, Fine-tuning and inference for this model, however still used 4096 and 2048 input lengths identical to other models. For all tasks, CHARFFRMER models used a downsampling rate of 2, while Byte-level TS+LASC models used downsampling rate of 4 (Clark et al. 2021). The downsampling rate of was picked by ablating the downsampling rate on the TyDiQA GoldP validation set. CHARFORMER models for XNLI and PAWS-X X additionally did not back- propagate into the GBST layer during fine-tuning. Checkpoints were picked based on the dev set metrics, and then evaluated on test set. Reported metrics represent the macro-average of all languages in the task.
which
batch size of 1024 and CHAR-
was prohibitively slow, see Table
16

---

Published as a conference paper at ICLR 2022
8.3 ABLATION STUDY
This section presents our ablation experiments for both English and multilingual tasks. We analyze the impact of various hyperrparameters and modeling choices such as using offsets 1D convolutions. Across experiments, we find that pre-GBST convolutions are prefemrd to enumerating offset blocks, it results in similar (or better) quality but more efficient implementation. For English tasks, block score calibration (BC) improves performance. We note that in the multilingual setting, block score calibration has little effect. The impact of different downsampling rates varies across tasks and model sizes. We also experimented with different convolution filter sizes in English and found that they did not significantly impact performance. Likewise, using a different character span corruption rate during pre-training did not significantly impact performance. Adding feed-forward layers to the CHARFORMER module in similar fashion to a Transformer block was also not obviously helpful.
Table 7: Ablation studies with CHARFORMER Small English tasks.
Ablation
d,
Size SST-2
MNL
IMDb
Offsets
2
89.11
79.50
90.49
Conv
2
S
89.11
79.65
90.63
Conv + BC
2
S
89.56
80.15
90.60
Conv + Offisets BC
2
S
89.11
79.68
90.48
Conv
3
S
89,45
80,07
90.15
Conv
4
S
89.11
79.82
90.21
Conv
2
B
90.60
82.92
91.46
Conv
3
B
91.40
82.74
91.46
Conv
4
91.40
82.67
92.33
Table 8: Effecc of freezing the GBST layer for XNLI and PAWS-X.
Model
d,
Freeze GBST
XNLI (Zerv)
XNLI (Translate)
PAWS-X (Zero)
PAWS-X (Translate)
CHARFORMER,
2
No
44.5
62.7
27.9
37.5
CHARFORMER
2
Yes
50.9
68.7
77.1
84.8
CHARFORMER,
Small
47.9
36.8
CHARFORMER
3
Yes
43.2
68.6
77.8
83.7
CHARFORMER
4
No
47.5
47.5
30.9
36.9
CHARFORMER
Yes
43.6
43.6
77.9
83.5
8.4 COMPARING DOWNSAMPLING APPROACHES
In Table TO we compare GBST downsampling with LASC downsampling (Clark et al. 2021 on
TyDiQA Goldd.. For this experiment we use the same byperparameters as in Section 3.3 except the pre-training input length is 1024 instead 2048. Note that this difference is negligible (0.1 FI) for CIIARFORMER Base, d, 2 which also appears in Table 4 All hyperparameters are fixed between CHARFORMER and Byte-level TS+LASC Following (Clark et al. 2021) we set d, = for LASC, and compare CHARFORMER the same downsampling rate. We additionally include d, 2 and d. = for CHARFORMER for comparison. With the same hyperparameters and downsampling rate, CHARFORMER outperforms lytellevel T5+LASC TyDiQA-GoldP.
Table 9: Effect of d, on TyDiQA-GoldP (in-language multi-task). Model
d,
TyDiQA-GoldP
CHARFORMER
69.6
CHARFORMER small
3
68.1
CHARFORMER small
4
66.6
Byte-level T5+LASCsmatt
4
64.9
CHARFORMERBacc
2
75.8
CHARFORMER Base
3
74.3
CHARFORMER Base
73.2
Byte-level TS+LASCRRaw
70.6
17

---

Published as a conference paper at ICLR 2022
8.5 SCALE EXPERIMENTS
In this section we report preliminary results for scaling Charformer using the same number of parameters mT5 Large and Large (1.23B). We follow model scaling configuration identical to ByT5 in these experiments, and use the same hyperparameter settings as our main multilingual results.
Table 10: Comparison on TyDiQA at 1.23B parameters. Due resource constraints, the Charformer result below uses 100K less pretraining steps than ByT5 and mT5.
Model
TyDiQA- GoldP FI EEM
mT5tarre Large
85.3/75.3
Large
87.7 79.2
CHARFORMER*
86.3 77..3
Results The CHARFORMER model under the same scaling as LyT5 was able outperform mT5 à very strong baseline. Our preliminary results at this scale shows that CHARFORMER is competitive with, but is 1.4 Fl behind LyTge However, we point out two important notes. First, the CHARFORMER result is undertrained compared to ByT5 Large since 10% of the pretraining has not finished. Second, the CHARFORMER model also twice as fast as ByT5, as seen from Table
8.6 MULTILINGUAL EXPERIMENTS
This section contains detailed results for our multilingual experiments.
Table 11: Compute metrics of base models at longer (2K) input length on the mC4 pre-training
corpus, using a batch size of 64 on -3 chips.
Model
L
d,
[0]
Speed (steps/s)
FLOPS
Byte-level T5
2048
1
200M
2.7
2.0 x 10
Byte-level TS+LASC Buse
2048
4
205M
5.5 x 10
CHARFORMER
2048
2
203M
6.1
9.5 x 10
CHARFORMER Base
2048
3
203M
10
6.5 x 10
CHARFORMER
2048
2
134M
6.1
9.2 x 10
Table 12: Per-language breakdown of in-language multi-task TyDiQA-GoldP results.
Model
bn
Ca
C0
500
52
Arg.
IBEERT Bu.. (Sabword)
129M
-1-
-5
73.6880
m/.3 (Subward)
582M
$4.2/11.8
80.0/99.0
76.8655.2
80.1/99.3
85.5/75.0
70.3661.
77.5161.4
83.6/74.9
88 18.2778.0
808/700
Byle level Tastase
200M
$1.3867.
66.8/56.6
69.8:59.5
75.6/63.0
81.6/2244
64.658.77
74.1/60.8
$1.3/4.33
85.076.1
75.6665.4
Byse-level 1SiLASC
205M
78.1662.3
61.1/504
66.7555.1
12.5504
79 79.9668.3
51.5433
704458.7
74.7667.5
80.27122
70.659.77
CHARFORMER
203M
$1.8/5799 9
69 1/80
71,48905
76 36.36422
83.0/73.1
62.7/54.3
74.7611.7
80.2/73.3
83.675.0
75.9665.6
CHAREORMERgpu
134M
$2.4/88.
1/6.3
75.4/4..3
79.5868.
85.0075.9
66.6158.0
77.0644.3
81.3/74.1
86.3/78.6
79 1768.8
CHARFORMERK Po
85.2/74.5
78.7667.3
76.865.9
81.97706
86.7/79.1
69.46616
79.2/67.1
833/05.2
88-320066
81,271133
Table 13: Per-language breakdown of translate-train-all all XQuAD results.
Madel
<1
bi
mm.5pum (Sabword)
582M
72.455.2
76.959
76.8258.
$3.1/703
79.091.2
71,45334
16.15
57
67.963..
72.551.4
75.956.3
36.099.7
75.3/59.7
T5
62.44555
3065542
38.352.3
2/2.222
71.081
57.782.77
CHARiOO
203M
65.7498
24.3300
111183.1
63.2705
778811
5434050
70.3530
76.950
62.096.
71//55.9
134M
20.35537
75.6814
34.455.11
85 19737
703085.
69.182.7
16 736.3
57.0512
73.05588
768877
67.480.
75.659.0
134M
72.655.0
79.0/223
1L99%.1
55 6.45
80,055.4
70.656.1
77.802.2
56.14992
76.058
71.3994
66.061.5
74.25598
18

---

Published as a conference paper at ICLR 2022
Table 14: Per-language breakdown of translate-train-all MIQQA results.
Model
[8]
de
zb
Avg.
mT5 Bosc (Subward)
582M
61.1/40.7
65.5/49.2
80.7/66.3
70.7/52.1
63.6/44.3
68.0/47.6
63.5/39.4
67.6/48.5
Byte-level T5 Buse
200M
52.6/4.2
60.5/46.1
77.7/64.8
67.149.2
52.9/6655
63.6/4388
58 38.3/36.4
61.8/44,4
Byte-level 13+LASCBure
205M
50.8/32.0
$8.1/43.5
75.8/62.2
64.7/46.7
49.2/32.6
60.4/40.4
52.6/30.6
58.84111
CHARFORMER
203M
53,5/34.5
61.3/46.8
78.5/65.4
67.2/49.3
54.5/37.6
64.3/43.9
58.8/36.6
62.6/44.9
CHARFORMER
134M
58.3/39.1
65.7/50.5
81.8/68.7
71.0/5311
57,7/40.8
67.3/46.8
62.7/40.8
66 3643/
CHARFORMER Foa
82.2/69.0
72.1/54.5
59.7/42.9
68.2/47.4
62.4/4077
67.2/49.
Table 15: Per-language breakdown of translate-train- all and cross-lingual zero shot XNLI results.
Model
10
bg
do
el
CS
fr
TU
SW
th
ur
vi
zh
Avg.
Translate- Train-All
mT (Subwond)
SK2M
74.4
76.5
77.7
76.1
$2.0
79.1
77.9
72.2
76.5
71.5
75.0
74.8
70.4
74.3
76.0
75.9
Byte-level
200M
71.0
70.6
76.9
14.0
73.4
63.7
66.2
65.7
69.4
62.8
69.6
69.0
69.4
Byte-level IS+LASC
73.4
CHARFORMER
203M
69.5
72.9
72.7
72.6
78.2
74.5
73.6
67.0
71.7
67.9
68.1
70.8
65.0
70.7
71.5
71.1
CHARPORMIR
134M
7088
75.7
75.9
73.1
809
76.9
76.8
65.6
74-7
65.7
67.7
72.0
63.1
72.9
71.5
72.2
CHARFORMER
134M
71.1
75.9
73.6
74.2
S0.8
76.6
76.8
69.2
72.2
68.2
71.0
71.2
05.7
72.9
73.0
72.8
Crots-Loguul Zero-Slot
Toar (Subward)
mm33ice (Subward)
SK2M
73.3
78.6
77.4
77.1
84.7
80.3
79.1
70.8
77.1
73.2
72.8
74.2
74.1
75.4
Byte-level 15
200M
56.7
61.2
63.0
60.9
79.2
70.1
43.9
61.0
45.5
43.5
52.0
44.3
58.3
57.4
Byse-leve IS+LASC
205M
53.3
58.8
62.2
54.9
77.1
68.6
65.4
44.7
58.4
46.1
43.6
50.4
42.8
55.9
46.1
55.2
CHARFORMER
203M
55.7
61.8
60.1
77.3
69.9
67.9
47.9
51.0
43.5
59.1
53.4
57.6
CHARFORMER:
CHARPORMERR Frene.ComgFP
134M
70.2
77.0
76.6
56.5
71.2
Table 16: Per-language breakdown of translate-train-al and cross-lingual zero-shot PAWS-X results.
Model
de
en
es
fr
ja
ko
zh
Avg.
Translate- Troin-All
mTSg (Subword)
92.5
83.6
Byte-level Ba
200M
89.3
94.6
90.1
90.3
81.4
81.1
82.3
87.0
Byte-level DS+LASCriam
205M
87.3
93.1
89.2
89.2
81.0
72.9
80.8
84.8
CHARFORMER Bavs
89.9
94.6
89.8
91.4
82.7
78.4
83.3
87.2
CHARFORMER BBos
134M
89.9
95.9
91.8
92.2
83,9
78,9
84.4
88.2
CHARFORMER Lomg
134M
90.7
95.1
92.2
92.2
84.1
81.6
84.6
88.6
Croxs Lingual Zero Shot
mBERT Base (Subword)
85.7
94.0
87.4
87.0
73.0
69.6
77.0
mT5 (Subword)
582M
89.4
95.4
89.6
91.2
79.8
78.5
81.1
86.4
Byte-level TS Bave
200M
84.7
93.8
85.8
86.4
72.2
67.9
75.2
80.9
Byte-level TS+L.SSCpssc
84.1
67.9
66.4
73.4
79.0
CHARROORER
203M
86.1
94.8
87.2
88.0
70.1
69.7
75.5
81.6
CHARFORMER
134M
89.6
95.2
90.7
90.7
77.1
74.4
78.9
85.2
Staee LongPT
89.8
95.3
88.7
89.7
74.5
78.9
83.7
8.7 EXAMPLE IMPLEMENTATION
For additional clarity, we include a simplified implementation of the GBST module in Tensorflow below. Default hyper-paramettrss here match those used in the paper.
Crom Lyping Suport Optional
ingort bensorflox as tf
layers
leyers
lase GRSTlayer (keres Tayers, Sopems.Teee [ayer))
eneper Forms Cha GBST sequence.
Attributes:
ingut ahhaps Shape [lwn, wibeddinggxiiee of Ingut Lansar in future calls,
downsemple e ratez Tht engeg of hou much to downeampl by.
19

---

Published as a conference paper at ICLR 2022
nax_subeord wicck Incegez block use for anunoration. biockLattention; thethez to use bleck seare calibration. wow.___ooringgntteeee. nodule ree poraneterizsd block seoring.
the the pre-oBST convolution kernel.
def init init__(ee1, (self,
inpot_sshape: tf.Ionsor, deenaanlla_cals: int wwL_ubbarrr_bloc_..aaa wiidca int 4,
siock_attontion: beol False, cong_kennel_saae OgLioral(iat]
aupez (asstaayes, -nni)_____t___
scir. ses..ddmmmmpll_rtt. sownsample_
self, an www.ee saarrconv_korre cia cony_kernel_siae
self.conv layer law__aayer..oom11:
shape (-1,, wedf.esw_jeeeeele sise, inpttalaapeinntt..ooo
self blokklaatttttttoon Diosk_actention
belf.blook.seoringg
benss Palse
daf calliseif, iapuzs):
sneper forms domnsappling on the charactee soale Eapus represintal ion.
Azgs:
inpubat float Penser of shape (bacchaaiee, leaght,, cnbedding_atzer
Belucme:
<float> (batch_size, Lenghh / dovnaanple_saze anbedding size). Downsampl ed seguences.
ene
length inputs. shape
If welf.comv www.e/keeeeleeeee
irpots selffoonv harpw_llyyrttiguree
all all_block =
all
subaurd_len rangell, selffmas..uhoonddbeee
iapded_iopat impats
Pad the sequence length if needed.
if Length 1 salmord_len [-
pad subword_len -inggh
ler)
padding iff.constantil [0, 0l, 10, ped ant 19, 01]1 indieed__oppt Lf.gadtinguts,, padiling)
erbeddings candidates shapes [batch, (uslen/subwerddee. din] scores
scores shsper (batch, len_sson/d condidates = tf.nn.avg. padeed_input, [sabword_lens scsides" leubvord_enn "VLI)) block_ecores block cet.bbloo____cinnn_ wwwring.neeeeeeeeee.o network deardidetes]
Upsemple back the original
seq re.repta reti led hwl.okoacooooeo.o tfurepeat abloek_ aubsard_len, exix=l)
axis=]]
Repad upsampled
if needed.
if biiiees_bioc__ccoooe
length:
repad ame length - ret:lld_pboee. wwcc__ooc__o_ooo btt.ddd___ecc_crresseooo saapess
repadding an (100, 0i, (8, repad ant (6, 0]13
retiled_ecq ts.paddrettiodd_ee zopaddinat
retiled bacile__bloo__ccoo___
tt.pas (retilee block_ocoroeeee repadding)
Bake sure everything is the right length and add nex dincnsion to concat candidate blocks
scoreals, flergeh, retiled_eeq retilet_reql; rlenghh, is Heae]
retfled
all alloolost_tee socres aspenees bto.eetreeeeedd...oo.oooo.o scoksss uul_eeuoncee.sppaddiees.
block_scores scores tt.concatiall block scores, axis=-1) block_scores s5.an.softmax poarek_ axise-1) cancidates to.concst (sll
block
exien-1)
1020: Black score calibration block-by-slock attenticn is anitted in this implementation. sizeh_size aun_osncidates length x din candidates candidatas octput t2.reduso_out poc.iiiccssssssssssss... sxis--1) bez length
Domsanple by mean pooling.
selr.demnsan
output tc.na.avy_u avp_pol
output, iself. wei...vvaampasseee.
strides (ece-ssel..... sowsaasssssaas..tttt.....
padsing=*
20


Reviews:
[{"review_id": "okAJFYfnf2R", "reviewer": "Reviewer_YTQy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n1. The GBST module is simpler and faster. It has on-par performance but is 35%-80% faster with less memory usage than current SOTA byte-level transformer ByT5. Comparing with the lighter ByT5+CANINE model, it has similar speed and memory usage, but more straight forward and has better accuracy. It also has on par performance and less computation costs in most tasks comparing with T5/mT5.\n2. The re-scaling of the model (SBase) achieves better performance with ~20% less computation. Though this idea to have deeper encoder is from previous work, it’s still informative to have the experiment.\n3. The paper is very well written and has enough experiments to show the model’s performance in accuracy and capability in different tasks. \n\nWeakness\n\n1. (minor) The technical novelty of this paper is only GBST, a dense layer over n-grams, while training setting follows ByT5, and the best SBase model is also an implementation of the deep encoder concept.\n2. The scalability of the model is not shown. All the experiment focusing on models (other than mT5) with a parameter scale of 200M. But the paper ByT5’s smallest model has 300M parameters and performs much better than with 200M parameters (comparing Table 4 of ByT5 and Table 4 of this paper).\n3. (minor) T5 seems much better than SBase under Zero-Shot settings, though it takes more computation. \n4. From the last sentence of 3.1 Baselines, it seems that the ByT5 model used for comparison is unscaled, and I couldn’t find in the paper about structure of this ByT5 (sorry if I missed that). This is important since in the ByT5 paper all the model variants have much heavier weight in encoders, and there is also an ablation study to prove that heavier encoder can improve performance a lot. In this paper, the scaled Charformer-SBase is also much better than the unscaled Charformer-Base. Thus it’s a bit strange that the best settings of both models are not used for comparison.\n\n", "summary_of_the_paper": "This paper focus on a gradient-based subword tokenization (GBST) method for byte-level transformers. The key idea is to use a linear transformation (with trainable parameters) + softmax to compute embedding for each character based on several n-grams in its neighborhood, and then use average pooling to downsample and decrease the sequence length.", "summary_of_the_review": "To summarize,\n\nPro\n\n1. Excellent paper writing.\n2. Extensive experiments.\n3. Simple but effective token-free module.\n\nCon\n\n1. Lack of experiment on different sizes of the model, especially performance when with a larger scale.\n2. The comparison seems not between the best setting (deep encoder, shallow decoder) of both baseline and this model.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T6p144l7TK-", "reviewer": "Reviewer_Md4g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper proposes a novel gradient-based tokenization module to address the issue of rigid subword tokenization algorithms and attains competitive performance. The proposed method saves engineering efforts for tokenization, which would be easy to generalize to different tasks. \n\n2. It is interesting to find that this module can deal with multilingual and noisy datasets to some extent, which gives credit to learning subword representations from characters. \n\n3. This proposed token-free module (GBST) has less computational cost than using the general methods of tokenization and can be well extended to other models.\n\nWeaknesses:\n\n1. The performance is not surprising, though the model indeed saves model size. It is not clear if the method works for other languages. For example, character-based models tend to achieve similar performance compared with word-based ones. Would this method has advantage over the character-based baseline then?\n\n2. The evaluations are based on base models. Since this model saves model size, would it achieve better performance compared with the same size (e.g., 200M like Byte-level T5 Base in Table 1)? Do you try large models? Besides, it would be impactful if this paper showed that GBST module could actually generalize in other kinds of models. \n\nMinor Comments:\n\n1. I wonder if average pooling is good way to down sample though it is really fast, but how about MLP? \n\n2. In Eq.3, the indices should be 0 to M-1 or 1 to M (total number of block size is M), i.e. $P_i = \\softmax([p_{0, i},  p_{1, i},  \\dots, p_{M,i}]$ -> $P_i = \\softmax([p_{1, i},  p_{2, i},  \\dots, p_{M,i}]$\n\n3. At Figure 1 (b), it seems the annotation `P5:8` in the fourth line is missed.\n\n4. In section 2.1.2, block sizes 1 < b < M should be 1 <= b <= M\n\n5. In section 2.1.5, “latent subwords $\\hat{X} = [\\hat{X}_i, \\dots, \\hat{X}_M]$  should be “latent subwords $\\hat{X} = [\\hat{X}_i, \\dots, \\hat{X}_L]$\n", "summary_of_the_paper": "This work aims to let the model learn the subword representations by itself from character-level embeddings of the input without tokenizing. A gradient-based subword tokenization module (GBST) is designed for replacement of the general tokenization. The proposed CHARFORMER, a re-scaled Transformer architecture (transformer with GBST) that integrates gradient-based subword tokenization, outperforms several byte-level baselines and performs on par with some subword-level models on three kinds of benchmarks, including GLUE, multilingual, and noisy text datasets. Moreover, CHARFORMER has the advantages of less memory usage, faster speed compared to the models with similar parameter size.", "summary_of_the_review": "This work is well-motivated. The overall design of the model is technically sound, and it provides moderate improvements on several datasets, while being memory efficient, faster, with fewer parameters. It is not clear if the advance of the method can generalize to other languages or larger models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PR-u2ksFT97", "reviewer": "Reviewer_NyMb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The work makes tokenization more adaptable by learning sub-tokens on-the-fly.\n+ Training speed and parameter efficiency are improved over more rigid, sub-word tokenization based Transformers in many cases.\n+ The tokenization method is based on a weighting of pooled byte n-gram embeddings. Score calibration and downsampling are technically sensible and the authors make a noticeable effort to ease optimization and not introduce foreseeably brittle complexities like extra hyperparameters.\n+ The n-gram weighting approach allows the sub-token weightings to remain somewhat interpretable.\n+ The downsampling operation allows an easy way of saving later transformer stack parameter size, which is shown to increase learning speed and reduce parameters.\n+ Potential limitations that require future work are mentioned proactively.\n+ Instances of reasoning about technical choices, e.g. \"narrower encoders\", provide instructive details and help make the foci of the experiments more deducible.\n+ Experiments are intentionally controlled (deconflated), to produce more robust insights.\n+ Table 2 uses AUC-PR rather than the original works AUC-ROC as performance measure, which can be considered an improvement.", "summary_of_the_paper": "This work first implements a byte-level (character) tokenizer, which is learned as part of the model rather than as a preprocessing task. To limit the computational burden of character level encoding on the downstream architecture, the method creates byte n-grams and combines them via a scoring network as outlined by the authors. Common Transformer architecture can use the tokenizer and even be narrowed. Finally the model is evaluated using multiple NLU and noisy NLP tasks. As a result, the approach achieves comparable end-task performance to sub-word and other byte-level Transformers, while improving parameter efficiency as well as ease of use for non English NLP tasks.", "summary_of_the_review": "This work first implements an end-to-end trainable byte-level (character) tokenizer, while limiting computational overhead via a learned token weighting. It is compatible with common Transformer stacks, but can reduce their width for computational gains, without sacrificing performance, and in some places gaining performance. The evaluation on multiple NLU and noisy NLP tasks demonstrates only slighly less to slightly stronger performance compared to byte and sub-word tokenization models. Ease of use, being conceivably language agnostic, training more efficiently and allowing easy reimplementation (pytorch), make this work a welcome contribution to the field.\n\nTherefore, I recommend accepting this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IXaW4xHllF4", "reviewer": "Reviewer_yH31", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- (Unclear) How is the proposed GBST different from convolution, despite the scoring part?\n- (Pro) Extensive evaluation results across many NLP classification tasks in mono- and multi-lingual settings.\n- (Pro) Charformer seems to achieve similar predictive performance to Byte-level T5 while being more computationally efficient (Table 6). \n- (Mixed) Charformer achieves worse predictive performance than subword models BERT and T5, which seems to suggest the proposed GBST gives little lift from character-level to subword-level. However, the Charformer models presented are smaller in size or faster in speed, or more efficient in FLOPS. Thus, hopefully, it will provide people with more choices for their use cases.\n- (Mixed) Experimental results do not show strong improvements on accuracy metrics. Table 1-3 shows that Charformer is generally on par with Byte-level T5. Sometimes Charformer wins; other times Byte-level T5 wins. Either with small differences. In the multilingual case (Table 4), we do see Charformer consistently outperforms Byte-level T5 with a small margin. In all cases, Charformer is significantly worse than subword models. The rescaled version Charformer_S is able to achieve comparable and sometimes better accuracy than subword models. However, the re-scaling is not tied to Charformer and can be applied to other character-level models as well. It is not clear if the gain is because of the proposed GBST or solely due to the re-scaling.\n- (Con) It is not clear if the proposed GBST module learns meaningful subword tokenization. Better qualitative evidence, e.g. examples that highly scored subwords align with human intuition, or quantitative evidence, e.g. how the learned tokenized subwords align with established methods, or how they are better at solving lexical tasks, would help.\n- (Con) Many technical details are unclear/confusing. It would help if the authors can provide more and clearer technical details. If limited by pages, the author can point readers to appendices. Make sure the main text still contains the necessary details when moving things to Appendices.\n- (Con) No code provided. Providing code will greatly help reproducibility and help clarify many technical details not fully described in the paper.\n\n### Questions to the Authors\n- How is the proposed GBST different from convolution, despite the scoring part?\n- Can the authors help clarify the design of GBST and implementation of Charformer? For example, what is the Transfomer stack used by Charformer? It seems like T5, but I don't find it mentioned anywhere. See other localized points below.\n- All the tasks in the paper seem to be classification tasks. Is Charformer encoder only? If so, how do the authors ensure a fair comparison with encoder-decoder models, e.g. T5? If not, is the decoder part of Charformer character-based, or it also uses some notion of subwords? \n\n### Localized Points\n- Eq (1). What is used as $F$ in the experiments?\n- In \"Considering Offsets\". How is 1D convolution applied to X? How does it save computation?\n- Figure 2. How to interpret this heatmap for block size > 1? How are they aligned? For example, we see a high score for block size = 3 at \"k\", is it for subword \"tok\" or subword \"ken\"?", "summary_of_the_paper": "The authors propose a soft gradient-based subword tokenization (GBST) module with the aim of improving tokenizer-free end-to-end training of language models. The GBST module takes a byte-level sequence and computes all possible subword representation (up to a length) in a convolution-like manner, the results are then pooled, scored, and weighted with other add-ons including 1D-convolutions and an attention-like position-wise score. The authors use the GBST module followed by an encoder-decoder Transformer stack similar to that of T5 and call it Charformer. The authors conducted experiments on a range of NLP classification tasks under monolingual and multilingual settings. The trend seems consistent that subword-level models > Charformer ≈ byte-level T5 on monolingual and multilingual clean data, Charformer ≈ byte-level T5 > subword-level T5 for monolingual noisy data. In terms of efficiency, the authors show evidence that Charformer uses fewer parameters and FLOPS than subword T5 and proceeds more steps per second than byte-level T5.", "summary_of_the_review": "A plausible proposal for the promising direction of learning subword tokenization end-to-end. However, some important design and implementation details are missing or confusing which are not helped by the lack of source code. The extensive experimental results themselves would benefit the community. However, they don't seem to strongly support the advantage of the proposed module in learning subword representation either in terms of end-task accuracy or explainability. If the authors could clarify what they did exactly with their GBST module and thus their novelty, I would be inclined to accept for the merit of extensive evaluation results and one more alternative that lies between subword-level and plain-byte-level models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-TjqMbPWTus", "reviewer": "Reviewer_zaHb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Avoiding the reliance of NLP models on tokenizers is an important problem. Tokenizers influence the input structure and the design of the model. The input structure is not amenable to graceful modifications after the initial training which can be limiting. \n2. The idea to calculate position representations based on a weighted average of subword blocks is interesting. This way the representation captures information from nearby n-grams. \n3. The experimentation has some competitive results and focuses on both performance and speed aspects. \n\nWeaknesses\n1. My main concern is that the proposed method does not seem to be satisfying the criteria for a proper tokenization method. \n     * First, it does not learn to segment the input to different chunks to be fed in the contextualizer but rather downsamples the sequence in a fixed way. The latent subword representations could be captured by combing multiple convolutions and a simple pooling function per position, which makes the method seem less novel.  \n     * Second, the soft selection at each position is performed over blocks of different size that correspond to that position and does not take into account global information e.g. by computing scores of all possible segmentations to maximize for the most likely one.  \n2. The current framing would suggest comparison to existing tokenization methods like BPE or recent  optimized ones to the downstream task [1,2] in a controlled setting (e.g. similar vocabulary size and contextualizer design), but this was not explored at all in the evaluation. In addition, there is no proper evaluation of the learned segmentations other than the visualization of the learned weights for a single example. \n3. The results are not that good compared to pure character-level models on average which makes results less exciting. Also, one of the claimed benefits of the proposed method is that is faster is mainly due to the fixed downsampling which is not something new and applies to other character-level baselines. \n4. Studies that focus on tokenization like the ones cited below evaluate on tasks where tokenization is important for reaching state-of-the-art performance like machine translation. It might be worth evaluating there.  \n\nQuestion:\n* In section 8.2 about monolingual datasets, was the downsampling rate optimized only for the proposed model?\n\n[1] https://aclanthology.org/2020.findings-emnlp.120.pdf\n\n[2] https://arxiv.org/pdf/2012.15671.pdf\n", "summary_of_the_paper": "This paper aims to remove the reliance of NLP models to external tokenizers. It proposes a gradient-based subword tokenization module that can be used in any neural model. The module scores predefined candidate blocks in a soft way for each position and then it weight averages them to obtain a mixture of subword representations. The resulting sequence is then downsampled in a fixed way to make processing easier. The proposed method integrated in a deep narrow transformer model performs on par and slightly better than character-level baselines on monolingual and multilingual classification respectively. In some settings it also outperforms strong subword-level baselines.", "summary_of_the_review": "The paper focuses on an interesting problem and has many experiments with state-of-the-art results in certain settings. My main concerns are regarding its novelty, framing, and evaluation.  Even though some of the results are competitive, the experiments do not provide enough evidence that the learned  tokenization is crucial for achieving the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a soft gradient-based subword tokenization module (GBST) that learns latent subword representations from characters. GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. The resulting model was tested on GLUE, and several cross-lingual tasks. The performance is competitive with ByteT5 and often similar to subword models while being more efficient in FLOPs and RAM.

Reviewers are mixed on this. The negative reviewer points to how this not being a real tokenizer and does not produce a tokenization, that experiments that use the base model do not address bigger scales, and that there is a lack of code which is important for this kind of work, and the resulting accuracy gains are not significant and the method being not interpretable. The positive reviewers like the extensive experiments, the efficiency improvements and flexibility / simplicity of the GBST module. The authors seemed to have addressed most of the reviewer issues by providing larger scale experiments and code. I believe the results are fairly strong, since one would not expect a big performance difference in a learned tokenization method, but rather efficiency or flexibility gains. The paper is generally well-written though details about the convolution should be included in the text (and not just the code).

Recommending accept.
Decision: Accept (Poster)