On the Effectiveness of Quasi Character-Level Models for Machine Translation
Authors: Salvador Carrión Ponz
Francisco Casacuberta Nolla
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep learning
Neural Machine Translation
Subword-level vocabulary

Abstract: Neural Machine Translation (NMT) models often use subword-level vocabularies to deal with rare or unknown words. Although some studies have shown the effectiveness of purely character-based models, these approaches have resulted in highly expensive models in computational terms. In this work, we explore the advantages of quasi character-level Transformers for low-resource NMT, as well as their ability to mitigate the catastrophic forgetting problem. We first present an empirical study on the effectiveness of these models as a function of the size of the training set. As a result, we found that for data-poor environments, quasi character-level Transformers present a competitive advantage over their large subword-level versions. Similarly, we study the generalization of this phenomenon in different languages, domains, and neural architectures. Finally, we conclude this work by studying the ability of these models to mitigate the effects of catastrophic forgetting in machine translation. Our work suggests that quasi character-level Transformers have a competitive advantage in data-poor environments and, although they do not mitigate the catastrophic forgetting problem, they greatly help to achieve greater consistency between domains.

Under review as a conference paper at ICLR 2022
ON THE EFFECTIVENESS OF QUASI CHARACTER- LEVEL MODELS FOR MACHINE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Machine Translation (NMT) models often use subword-level vocabular- ies to deal with rare or unknown words. Although some studies have shown the effectiveness of purely character- based models, these approaches have resulted in highly expensive models in computational terms. In this work, we explore the advantages of quasi character-level Transformers for low-resource NMT, as well as their ability to mitigate the catastrophic forgetting problem. We first present an empirical study on the effectivenesss of these models function of the size of the training set. As a result, we found that for data-poor environ- ments, quasi character-level Transformers present competitive advantage over their large subword-leve versions. Similarly, we study the generalization of this phenomenon in different languages, domains, and neural architectures. Finally, we conclude this work by studying the ability of these models to mitigate the ef- feets of catastrophic forgetting in machine translation. Our work suggests that quasi character-level Transformers have a competitive advantage in data- en- vironments and, although they do not mitigate the catastrophic forgetting problem, they greatly help to achieve greater consistency between domains.
INTRODUCTION
Neural machine translation (NMT) has become the dominant paradigm in the field machine trans- lation due to the impressive results achieved with encoder-decoder architectures (Sutskever et al. (0014); Cho et al. (2014)); Wu et al. (2016); Vaswani et al. (2017))
Despite these advances, the translation of rare unknown words has turned out to be more difficult problem to tackle than was initially thought. For which reason, authors have proposed different approaches to deal with this problem, which can be grouped into three categories: Models based on character-level representations. ii) Hybrid NMT Models. iii) Models based on word segmentation.
Character- based models are the direct approach as they naturally deal with
unseen words.
However, these models have historically resulted in unsatisfactory results (Vilar et al. (200(200)); ) Neubig
al. 2013) highly expensive models in computational terms (Luong & Manning (2016b)) To balance the benefits of word- and character- based models, Hybrid NMT models were invented (Lu- ong Manning (2016a)). The idea behind these models is to translate mainly the word level and only query character components for rare words when necessary. However, these models tend to be a bit cumbersome as they generally need two models to do the back-off. Finally, word segmentation approaches such as BPE (Sennrich et al. (2016)), Unigram (Kudo) (2018), or SentencePiece (Kudo) Richardson 22188), emerged to address many of the problems mentioned above, by allowing the encoding of rare words using vredbvocabulary of subwords.
Despite the success of word segmentation models, is still not clear how to determine the vocabulary size. The most common strategy is to choose arbitrarily large value (i.e. 32k), under the premise that large vocabularies tend to work well (Gowda & May (2020)). Despite this heuristic, we think that there must be a better way of choosing this value depending the dataset properties.
Motivated by these ideas, we decided to study whether NMT character-basee models had any advan- tages compared to models with large subword vocabularies (i.e 32k tokens). We theorize that since words follow a Zipfian distribution, models with large vocabularies would require large datasets to learn the long-tailed distribution of subwords. In contrast, chasecter-based models would be able

---

Under review as a conference paper at ICLR 2022
to learn it more easily in data-poor environments as their subword distribution is less pronounced than the distribution for large vocabularies. One of the main criticisms of character- models is that they involve longer sequences and therefore, more long-term dependencies. To alleviate some of these problems, we decided to study quasi character-level models, which are essentially small subword-level vocabularies (i.e 100-400 tokens).
Furthermore, we decided to study if character- models can mitigate the effects of catastrophic forgetting based on the idea that, if this phenomenon depends on the vocabulary's domain, when vocabulary is shared between domains, these effects should be mitigated.
The contributions of this paper are fourfold:
Quasi character-level Transformers seem to consistently outperform large subword-level Transformers low-resource environments. (Section 5.I.I.
When LSTMs and CNNs used, this phenomenon is still present but it is more sensitive on the chosen hyperparameters. (Section 5.1.3.
Quasi character-level Transformers appear to be more susceptible catastrophic forgetting than their large subword-level counterparts. (Section 5.2).
However, quasi character-level Transformers achieved better cross-domain consistencies.
(Section 5.2
2 RELATED WORK
Character-based models have been well-studied in the Natural Language Processing (NLP) field deal with the open-vocabulary problem. One of the first character-based models was proposed by Vilar et al. (2007). who treated the source and target sentences as a string of letters. Similarly, Neubig et al. (013)) viewed translation as single transduction between character strings in the source. However, despite the relevance of their contributions, their results were not satisfactory as their models generally performed worse than their word-hased counterparts.
Consequently, authors began to propose strategies to get the best of word- and character-level mod- els. These approaches were mostly based on Hybrid NMT models or subword-leve models with vocabularies built through word segmentation techniques. Luong & Manning (2016a) achieved open vocabulary NMT through hybrid models, which translate at the word level and consult the character components only for rare words; and |Sennrich et al. (2016) developed model segment words based on the Byte-Pair encoding compression algorithm proposed by Gage (194).. Later, more word segmentation models were proposed based on this idea, such Unigram by Kudo (0018)
SentencePiece by Kudo & Richardson (2018).
Chung et al. (2016) demonstrated that NMT model with character-based decoder can outperform NMT models with subword-level decoders. Similarly, Luong & Manning (2016b) and Costa-jussä & Fonollosa (2066) showed that competitive purely character-based NMT models were possible. Despite their relative success, these models were extremely slow to train and at runtime.
Recurrent neural architectures have been the dominant neural architecture for NLP tasks (Cho al. (2014)); Wu et al.) (2016); Sundermeyer et al. (2014)). However, in recent years we have seen convolutional neural models outperform recurrent models on some NLP tasks. Gehring et al. (2017) proposed a competitive sequence to sequence architecture based entirely on convolutional neural networks, and Conneau et al.] (2016) achieved state-of-the--ar machine translation with a purely character-level NMT model via deep convolutional stacks.
Recently, the attention mechanism (Bahdanau et al. (2015): Luong et al. (2015) along with the Transformer architecture by Vaswani et al. (2017), ) has led to significant progress in the field of NLP, due to the parallclizable nature of the Transformer architectures and its ability to deal with long-term dependencies.
This paper ends with a brief discussion on the ability of quasi character- based Transformers to mitigate the catastrophic forgetting problem in machine translation. As far as know, this is the first work to address this issue from the perspective of (quasi) character-level models, since most of the existing alternatives based on regularization (i.e. LwF by Li & Hoiem (2016), EWC
2

---

Under review as a conference paper at ICLR 2022
by Kirkpatrick et al. (2016)), dynamic architectures (Rusu al. (2016); Draclos al. (0016)
Complementary Learning Systems (CLS) and Memory Replay (MR) Kemker & Kanan (2017 7);
Lopez- Pozez-Paz & Ranzato 2017
NEURAL MACHINE TRANSLATION
3.1 NEURAL ARCHITECTURES FOR MACHINE TRANSLATION
The goal of any translation system is to transform an input sequence in given language into an output sequence in a target language.
Nowadays, this is usually done using neural models based on the encoder-decoder architecture; also know as seq2seq models in the machine translation community (Sutskever al. (2014)). The encoder part transforms the input sequence into internal representation, and then the decoder transforms this internal representation into the output sequence.
In this work, we focus study the Transformer architecture, although recurrent and fully con- volutional architectures are also explored for completeness.
Recurrent architectures (RNNs) were the first to be successfully applied in an encoder-decoder setup for machine translation. Even though there are many types of RNNs, most of them chain sequen- tially series of unit cells to process temporal sequences. We decided to use LSTMs (Hochreiter & Schmidhuber (1997)) because their units cells are explicitly designed to deal with long-term depen- dencies.
Convolution- based architectures (CNN) do not contain any recurrent elements. They can do this, because the idea behind this architecture is that the convolutional filters can slide through the quence of tokens, from beginning to end (Gehring et al. (2017)) Initially, the convolutional layers were intended to operate on 2-dimensional images so, order to apply them to data, have to imagine the sequence of tokens as if it were a Ixm image, where the height of the filter is equal to and the width equal to k. Being the number of consecutive tokens that the filter can observe at given time.
Finally, we have Transformer architecture by Vaswani based entirely on the concept of attention (Bahdanau et al. (2015); Luong et al.) (2015)) to draw global dependencies between the input and output. This architecture does not use any recurrence or convolutional layer to process temporal sequences. Instead, it makes use of linear and normalization layers, residual connections, attention mechanisms, and some tricks to encode temporal information.
(2017). state-of of ttthhhhan art architecture
3.2 THE OPEN VOCABULARY PROBLEM
In the written language it is common to find alternative spellings (i.e. color-colour) and typos (i.e scknowledge-acknooleee) that slightly modify the spelling of word but do prevent us, the humans. from knowing its meaning. However, if a model is using word-level representation, it will stop knowing a known word at the very first moment that is slightly modified and this modification is not in its vocabulary. Furthermore, it must be taken into account that most languages make use of agglutination and compounding mechanisms to form new words, which in itself makes it very inefficient to build vocabulary with all possible words of a language.
Because of this, researchers have proposed multiple approaches to deal with the open vocabulary problem. These approaches can be mostly grouped into three categories: i) Character- models ii) Hybrid models iii Subword-baa models.
Character-basee models contain the minimum possible vocabulary with which to form all possible words. Consequently, these models can translate rare or even unseen words tharacter-by-charade but at the same time, these models tend to be much slower and harder to train than word-based models. This is because they have to deal with longer sequences and therefore, longer long-term dependencies.
Hybrid models tend to translate primarily at word-level but fall back to character-level translation when rare unseen word appears.
3

---

Under review as a conference paper at ICLR 2022
Finally, subword-bare based models are fixed close-vocabularee that subwords instead of full words. These models allow us to represent infrequent or even unknown words as a sequence of subwords, but without the performance and memory issues that typically imply character-basel models.
In this work, we always work at the subword level but distinguishing two types of subword vocabu- laries. The first one is the quasi character-level vocabulary with 100-400 tokens, and the second one a large subword-level vocabulary with around 32,000 tokens.
To build subword-level vocabularies chose word segmentation model known as Byte-Pair Encoding (BPE) by Sennrich et al. (2016). We did also consider Unigram (Kudo (2018) and Sen- tencePiece (Kudo & Richardson (2018) but eventually, we decided use BPE due its popularity. The idea behind BPE is to start with a set of characters and iteratively extend it with the most frequent n-grams pairs in the corpus.
EXPERIMENTAL SETUP
4.1
DATASETS
The data used in this work comes mostly from the WMT tasks. In table we find all the datasets used in this work.
Table 1: Datasets partitions
Dataset
Train
Val
Test
Europarl v7 (ES-EN)
1.9M 100k / 50k
5000
5000
Europarl v7 (DE-EN)
1,8M / 100k / 50k
5000
5000
Europarl v7 (CS-EN)
635k / 100k / 50k
5000
5000
CommonCraw Crawl (ES-EN)
1.8M 100k
5000
5000
SciELO (ES-EN)
575k
120k
4961
4961
SciELO (PT-EN)
116k
3738
3738
NewsCommentary (DE-EN)
357k / 35k
5000
5000
IWLST'16 (DE-EN)
196k
993
1305
Multi30K (DE-EN)
29k
1014
1000
All values in this Table indicate number of sentences.
Europarl contains parallel sentences extracted from the European Parliament website; Common- Crawl consists of parallel texts mined from the Common Crawl dataset; SciELO is a parallel corpus made up scientific publications for the biomedical domain (health and biological); NewCommen- tary contains political and economic commentaries, crawled from the website Project Syndicate; Multi30k small dataset from WMT 2016 multimodal task, also known as Flickr30k; and the IWLST' 16 dataset comes from the IWSLT 2016 TED talk translation task.
In addition to this, we created two additional versions for each training set, with 100k and 50k sentences (randomly sampled) [
The validation and test sets were left untouched.
4.2 TRAINING DETAILS
First, we normalized and tokenized each dataset with Sacremoses, a Python port of the Moses to- kenizer by Koehn et al. (2007). Then, we created two vocabularies for each dataset with different subword granularities. The first one was made up of 32k subwords, while the other was limited to 100-400 subwords (characters common subwords P Both were built using FastBPE, tool to learn and apply BPE codes. The large subword-level vocabulary was achieved using 32k merge operations, while the small subword-level vocabulary only had 64 merge operations.
We used Fairseq v1.0...... 1.000a0 et al. (2019) as our sequence modeling toolkit. For our experimenta- tion, we use the following base architectures: LSTMModel, FConyModel and TransformerModel]
Later added more versions (120k, 35k, etc) perform additional experiments
"hhe exact amount tokens depended on each dataset
Long Short-Term Memory model (T (I.STMModel), Fully convolutional model (oCel) and the Transformer model (TransformerModel)
4

---

Under review as a conference paper at ICLR 2022
In addition this, tried two versions of the Transformer architecture: the Standard Transformer with 93M parameters and a smaller version with 28M parameters (see Table 2). Interestingly, both versions performed quite similarly in terms of accuracy (#1 BLEU). Also, the smaller version was notably faster than the standard version, so we end up using the smaller version for most of our experimentation.
Regarding the comparison of the models, we tried to make the comparison as fair a possible by only comparing models with similar number of parameters (see Table Similarly, we shared as many of the training hyperparameters as possible in order to reduce training biases. Nonetheless, this will be discussed in detail in see Section [5..3] as this comparison is tricky due to the impact of the chosen hyperparameters on the final result.
Table 2: Neural architectures
Model
Parameters
Hyperparameters
LSTM
6.8M (S) 27.8M (L)
layers 256 emb / 256 hid / bidir+attn 6x(256, 3) (enc/dec)
Fully CNN
6.5M (S) - 27.4M (L)
Transformer small
4.1M (S) 25.0M (L)
3 layers / heads / 256 dim / 512 finn layers / 8 heads / 512 dim / 2048 ffnn
Transformer standard
44.9M (S) (5)-92.11 92.7 (L)
Loss function: CrossEntropy (without label smoothing). Optimizer: Adam (Kingma Baj (015)) or NAG. Batch size: 4096 tokens/batch, Clip-norm: 0.1. Maximum
Common hyperparamters:
epochs: 100 epochs with early stopping. Beam width: 5.
Note: S/L stands for Small/Large subword vocabulary
4.3 EVALUATION METRICS
NMT models are generally evaluated qualitatively and quantitatively. The qualitative part carried out by humans, while the quantitative evaluation is carried out with automatic metrics such as BLEU Papinnni al. (0022), chrF (Popović (2015). or TER (Snover et al. (2006)).
These automatic metrics compute the quality of model by comparing its output with reference translation written by human. To do so, we computed the BLEU, chrF, and TER scores using Sacrebleu (Post 201 8 with its default arguments
RESULTS
5.1 ON THE EFFECTIVENESS OF CHARACTER-BASEE BASEDD MODELS
In this section, we propose series of experiments to demonstrate that under a series of conditions, quasi character-based models can obtain significant advantages over models based on large vocabu- laries.
Words follow Zipfian distribution, which implies that in large enough corpus, the most frequent word will occur twice as often as the second most frequent word, three times as often as the third most frequent word, and so Starting from this idea, theorized that models with large sub- word vocabularies would require large datasets to learn the long-tailed distribution of subwords (see Figure J. In contrast, thought that models with small subword vocabularies would learn their subword distribution more easily in data-poor environments than their counterparts because since the long-tail distribution of small subword-level vocabularies is shorter, the average relative fre- quency per token is should be higher which would lead to more opportunities to learn each token in more homogeneous distribution. Similarly, we argue that the advantages of character-based models should be lost when there is enough data from which to learn the low-frequent subwords of the tail due to the issues related to the longer tail dependencies of character-based models.
'Even though we evaluated each model using BLEU, chrF, and TER, all of them hehaved extremely simi- larly, we end up reporting just the BLEU metric for clarity.
5

---

Under review as a conference paper at ICLR 2022
Large subword vocab distribution (32k tokens)
Frequeccy
Quasi charactor-levd vocab distribution (350 tokens)
Frequency
Figure I: The frequency of the first 100 tokens of large subword vocabulary (32k tokens) and small subword vocabulary (350 tokens). The first distribution has a much longer tail than the second distribution, which translates to lower average relative frequency per token (see first Figure).
5.1.1 ADVANTAGES OF TRANSFORMERS FOR DATA-POOR ENVIRONMENTS
To test our hypothesis, we created smaller versions of our datasets by limiting the amount of training data to 100k and 50k sentences. As expected, we see in Figure 2 that as we limit the size of training set, the performance of quasi character-level Transformer increases compared to their large subword-level versions. The right column corresponds the full datasets for the Spanish-Engle German Engllish, and Czech- Each-- pairs, while the middle and left columms correspond to the same datasets but limited to 100k and 50k sentences. Our experimentation shows that for each of the original datasets, the Transformers with large subword vocabularies outperform every quasi character-level Transformer by around 5 points in BLEU, However, when trained them with the smaller versions of those datasets (middle and left columns), every quasi character-level Transformer outperformed each of their large subword-level versions.
Europar 1-M (es-an)
Eurepad-100k (es-en)
50
Europar (ss-en)
39.6
0
35,2
31.7
24.4
20
20
sulwwrd Quas: chan vocab
subwors
Large subwond vocab Quaui chan-
Quasi chan-leuel
Dumpari
Europart Semal
Eumpert Sma
(2M) esver)
[100K; en-ent
(558: esee))
Eumpari-MM (deven)
Europark- bock (dwwnn)
Fumpari-50k (dewan))
2
24.0
20
20
16.6
Largs subword vocub
Quisi tha level vicah
10
Large suceord vocab Quasi chanieuel voras
10
Large subword vocab Guavi chae- zoe vocah
0
Pumpari Lange
Europori Small
Eumport Small
Europart-630k
Europar-100e
Europarl-Sck
42.5
37.1
31.0
30
20
370
20
Quas: charlevellwee valab
Large cubword vocwo
10
Quasi char lovel vocab
Large subeorr vocwb
10
Lange subwent varab Quadi char level vocab
0
Buropari Lange
perspart Small
Eunoput Sma1
Figure 2: As limit the training data (left to right), we that quasi character-level Transformers performs better than their large subword vocabularies versions. Similarly, this phenomenon seems to occur regardless of the language (top bottom).
5.1.2 LANGUAGE AND DOMAIN GENERALIZATION
Initially, we performed the previous experiment on the Opanish-Englice pair, but in order to account for potential language biases, we decided repeat the experiment but using different languages.

---

Under review as a conference paper at ICLR 2022
Our go-to alternative language was German since it is well-known for its propensity for nominal compounding and agglutination. Similarly, we added the Czech- Basch-English pair to our experimentation as it keeps more distance with the other two pairs (see Figure 2) as well other pairs for com- pleteness (i.e. Portuguese-Fnnglish Spanish-Fngglsh), This phenomenon was consistent across all language pairs tested, which could imply that this is language-agnostic phenomenon. Nonethe- less, we would like to repeat these experiments with more distant pairs (c.g. Chinese-Russian, Japanese-Spanish, etc.) that we could confidently assert that this is indeed language-agnostic phenomenon.
Since all the previous datasets came from the Europarl domain. we wanted to explore whether this phenomenon could be caused by domain bias. To do so, we repeated the same experiment but on different domains, such as crawled data, political and economic news, health and biological sciences, transcripted talks, and multimodal transcriptions. These domains came from the following datasets: CommonCrawl, NewsCommentary, SciELO (health and biological datasets), IWLST' 16, and Multi30k. See Figure B
CommonCrrun
ScIELD (Health domain)
50
42.6
35.6
35.2
40
26.5
22.6
BLE
28.7
20
20
Large subwerd vocab
Large subwerd vocab
10
Large subword
charloord vocab
char-lond vocab
Quasi charlleed vocab
CammannConn Crom Large
Cammontruss
Large 1573K)
Blomesica Small
NexsCarmertary Large esiert)
Canmaramrr Small
Figure 3: The advantages of quasi character-level Transformers in data-poor environments are sistent when models are trained on different languages and domains. From left to right we have the results from models trained the CommonCrawl, SciELO, and NewCommentary datasets, which belong to different domains. In each of those figures, we see that when the training data is lim- ited (two columns on the left of each figure), quasi character-level models outperform their large subword-level counterparts, regardless of the domain.
As we see in Figure 3 the phenomenon here studied is present when we change the training domain crawled data (CommonCrawi), scientific literature (SciELO), and, political and economic com- mentaries (NewsCommentariess) In addition to these domains, we also experiment with transcripted TED talks (IWLST 16) and image descriptions (Multi30k). As expected, quasi character-level mod- els kept outperforming their large subword-leve' versions when we limited the training sets. They improved the BLEU score in 6.2pts for the IWLST' 16 dataset and 2.3pts for the Multi30k dataset.
5.1.3 NEURAL ARCHITECTURES GENERALIZATION
As all the previous studies were done using the Transformer architecture, we decided to study whether the phenomenon described above generalizes to other neural architectures for machine translation, such LSTMs CNNs. Specifically, we focused our research on bidirectional LSTMs with attention mechanisms and fully convolutional architectures like the one described in Gehring al. (0177). To have fairer comparison, we only compared models that had similar number of parameters for given vocabulary (i.e. 25-30M parameters for vocabularies of 32k subwords).
Since the performance of model does not depend solely on the number parameters but on its architecture, how it was trained, etc. We decided to standardize the training hyperparameters we used amongst all models as much as possible. addition to this, we performed multiple runs for each model with slight variations in their hyperparameters and initializations to account for this performance variability.
From experimentation, observed that expected, when large subword-level models were trained with enough data (unlimited training sets), they outperformed their quasi character-level counterparts. This happened for all three architectures (LSTM, CNN, and Transformer). How- ever, when the training sets were limited to 100k or 50k sentences, quasi character-based models performed better than their large subword-level versions in most cases (see Figure
When Transformers were used, the ones with quasi character-level vocabularies consistently out- performed the ones with large subword-level vocabularies. In contrast, when LSTMs or CNNs,
7

---

Under review as a conference paper at ICLR 2022
their performance was dependent their hyperparameters. In the case of LSTMs, the quasi character-leve versions always end up outperforming the large subword vocabularies, but the im- provements not impressive as the ones with the Transformer architecture. Finally, found CNN-based architectures to be particularly sensitive to the chosen hyperparameters, so the advantages of quasi character-level CNNs are not clear Nevertheless, with enough computational time and the right hyperparameters, quasi character-CNNs probably end up outperforming consistently their large subword counterparts.
us compared the other models.
Transformer anch cchttcttrr vocit
CNN architcture
Large sabeect
Figure 4: The green lines refer the best and worst runs of the models with large subword-level vocabularies, while the blue lines refer to the best run of the quasi character-level models. In the left figure, we see that quasi character-level Transformers consistently outperform the ones with large subword vocabularies. Then, in the middle figure we see that for the LSTMs, this phenomenon is not as strong as with Transformers. Finally, in the right figure, we can see how quasi character-CNNs do not present these advantages as clearly as in the other models.
5.2 EFFECTS OF QUASI CHARACTER-BASHD TRANSFORMERS ON THE CATASTROPHIC FORGETTING PROBLEM
In this section, we study whether quasi character- baased Transformers can help to mitigate the catas- trophic forgetting problem, whereby neural networks forget previously learned information after learning new information.
To do this, we have designed experiment in which we train model in domain A and evaluate domains A and to establish the baselines. Next, we fine-tune the model trained domain A with data from the new domain B, and then, it is evaluate it in domains A and B. In theory, the model trained domain A should perform well the domain A, and poorly in the unseen domain B. Similarly, after fine-tune it domain B, it should perform worse in A and better in domain B than the original model trained only on domain A.
In Figure Ba obtained BLEU of 33.3pts on its domain (health) and a BLEU of 14.3pts in the other domain (biological). Then, when fine-tune this model on the Biological domain (SciELO), the BLEU obtained on this domain increased from 14.3 to 31.7pts, while BLEU for the health domain fell from 33.3 10tttt. In Figure5b we see that something similar happened for the large subword- vocabulary, but the effects of the catastrophic forgetting problem were not as strong as in the other model. because the BLEU score went from 28.7 28.0pts.
that quasi character-level Transformer trained on the health domain (SciELO)
Vecabulary size: -350
Vocabulary size:
31.7
BLEU
28.7
21.0
23.0
14.3
Health Biological
16.0
Health
Biological
Health [Voc. domain=H)
H-B (Voc. domain=H)
Health (Voc. domain=H)
H-B (Voc. domain=H0
(a) Quasi character-level vocabulary
(b) Large subword-level vocabulary
Vocabularies seem to have strong impact on the catastrophic forgetting effects. While the quasi character-level model lost 12.3pts (see Figure the large subword-leve. model only lost 0.7pts (see Figure
Figure 5:
8

---

Under review as a conference paper at ICLR 2022
infer that vocabularies seem to have strong impact the effects of catas- trophic forgetting because character- vocabularies seem to make models more susceptible to the catastrophic forgetting problem than large subword-leve vocabularies.
From Figure
To further explore this problem, we repeated the previous experiment but taking into account the vocabulary domain. As result, discovered that the vocabulary domain has stronger impact on the model's 's performance than we thought. As shown in Figure quasi character-level Trans- formers present a high consistency between domains, while large subword-level Transformers seem particularly sensitive to their vocabulary's domain, to the point of achieving opposite results between domains (see right column of the Figure
50
Health
31.7
14.3
21.0
20
24.5
25.0
Mealhh
10
Woc. domainww)))
(Wts domannen
(ee. domainet)
(Poz. coma nwett)
(Wb domainett
(Vec domaineet)
Voc size: 350 tok: Voc. domairs Blologicca
Voc. sib
32% aak] Vec. domain: Blological
37.1
40
$1.2
33.7
:00
162
10.2
Heaft Billelis
Health Biwiggial
Healts
Biregiiall
Hi-6
Tealts
Blo cal
Hi-8 domtiin-8
Figure 6: Transformers with quasi character-level vocabularies (left figures) appear to be more con- sistent between domains than Transformers with large subword-level vocabularies (right figures)
Even though quasi character-level Transformers achieve better consistency across domains, they suffer more severely the catastrophic forgetting problem than Transformers with large subword vo- cabularies. We believe that by using specially designed regularization techniques address this issue such as LwF (Li & Hoiem (2016) EWC (Kirkpatrick et al. (2016)) these problems could be mitigated, leading more robust and consistent models.
CONCLUSION
In this paper, we have empirically studied the effectiveness of quasi character-level models and their ability to tackle the catastrophic forgetting problem.
Our studies reveal that for low-resource scenarios, quasi character-level Transformers are superior terms of accuracy compared to Transformers with large subword vocabularies. Although this phenomenon seems generalize with language and domain, also appears to be dependent neural architecture. When LSTMs were used, the improvements were not as impressive as with the Transformer architecture, but when experimented with CNNs, we noticed that the outcome was more dependent on the chosen hyperparameters than with the other models.
Finally, we showed that even though quasi character-level vocabularies do not to mitigate the effects of the catastrophic forgetting problem, they achieved a higher consistency between domains, which might lead to substantial improvements if regularization techniques applied to deal with catastrophic forgetting.
ACKNOWLEDGMENT
Work supported by the Horizon 2020 European Commission (H2020) under the SELENE project (grant agreement no 871467) and the project Deep learning for adaptive and multimodal interac- tion in pattern recognition (DeepPattern) (grant agreement PROMETEO/2019/22 acknowledge the support of NVIDIA Corporation with the donation of a GPU used for part this research.
We gratefully

---

Under review as a conference paper at ICLR 2022
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, 0e////4990473. 2015.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: decoder approaches. In Proceedings of SSS7-8, Kighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, Association for Computational Linguisties, 2014a.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on EMNLP, pp. 1724- 1734, 2014b.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. character-level decoder without explicit segmentation for neural machine translation. CoRR, abs/1603.06147 2016.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann LeCun. Very deep convolutional net- works for natural language processing. CoRR, abs/1606.01723 2016.
Marta R. Josta-jussà and José A. R. Fonollosa. Character-based neural machine translation. In Proceedings of the 54th Annual Meeting of the ACL (Volume 2: Short Papers), pp. 357-361, 2016.
Timothy Draelos, Nadine E. Miner, Christopher Lamb, Craig M. Vineyard, Kristofor D. Carlson, Conrad D. James, and James B. Aimone. Neurogenesis deep learning. abs/1612.033700, 2016.
CoRR,
Philip Gage. A new algorithm for data compression. C Users J., 12(2):23-38, 1994. ISSN 0898- 9788.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, 17, pp. 1243-1252, 2017.
Thamme Gowda and Jonathan May. Finding the optimal vocabulary size for neural machine trans- lation. Findings of the ACI: EMNLP 2020, pp. 3955-3964, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735- 80, 1997.
Ronald Kemker and Christopher Kanan. Fearne:: Brain-inspired model for incremental learning. CoRR, abs/1711. 100663, 2017.
Diederík Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting neural networks. CoRR. abs/1612.00796, 2016.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer. Ondřej Bojar, Alexandra Constantin, and Evan Herbst, Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL '07, pp. 177-180, 2007.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. Proceedings of the 56th Annual Meeting of the ACL (Volume 1: Long Papers), pp. 66-75, 2018. 2018.
10

---

Under review as a conference paper at ICLR 2022
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 November 4, 2018, pp- 66-71, 2018. doi: 18855/v1/d18-20111
Zhizhong Li and Derek Hoiem. Learning without forgetting. CoRR, abs/ 606.09282, 2016.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continuum learning. CoRR, abs/1 706.08840, 2017.
Minh-Thang Luong and Christopher D. Manning, Achieving open vocabulary neural machine trans- lation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the ACL (Volume I: Long Papers), pp. 1063-1063, 2016a.
Minh- Minh-Thang Luong and Christopher D. Manning. Achieving open vocabulary neural machine trans- lation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the ACL (Volume I: Long Papers), pp. 1054-1063, 2016b.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches attention-based neural machine translation. In Proceedings of the 2015 Conference on EMNLP, pp. 1412-1421, 2015.
Graham Neubig, Taro Watanabe, Shinsuke Mori, and Tatsuya Kawahara. Substring- machine translation. Machine Translation, 27(2):139-166, 2013. ISSN 0922-6567.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. CoRR, abs/1904.01038, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting ACL, ACL '02, pp. 311-318, 2002.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth WMT, pp. 392-395, 2015.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, 2018.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1600.04677 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the ACL (Volume 1: Long Papers), pp. 1715-1725, 2016
Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the AMTA: Technical Papers, pp. 223-231, 2006.
Martin Sundermeyer, Tamer Alkhouli, Joern Wuebker, and Hermann Ney. Translation modeling with bidirectional recurrent neural networks. In Proceedings of the 2014 Conference EMNLP, pp. 14-25, 2014.
llya Sutskever, Oriol Vinyals, and Quoc Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), NIPS, volume 27, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st NeurIPS, NIPS' 17, pp. 6000-6010, 2017. ISBN 518810964994

---

Under review as a conference paper at ICLR 2022
David Vilar, Jan-T. Peter, and Hermann Ney. Can translate letters? In Proceedings of the Second WMT, StatMT '07, pp. 33-39, 2007.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, 0bs/1609.08144, 2016.
12


Reviews:
[{"review_id": "r6PZ0qIjmVa", "reviewer": "Reviewer_K79J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to read and follow. However, I think this paper is missing a crucial comparison, of using character-based NMT to compare against the proposed method. Also, the vocabulary size of 350 feels arbitrary, similar to your point of \"32k vocab size being arbitrary\". I'd suggest having some experiments trying different vocabulary sizes and comparing the quality.\n\nMoreover, the authors criticize the character-based NMT to be slow and inefficient. However, none of the experiments is designed to benchmark the efficiency. I.e. comparison of the average tokens after tokenization with standard subword vs quasi-character vs character-based. I think this experiment will give a good insight into the performance between approaches.\n\nYour claim about \"it is still not clear how to determine the vocabulary size.\" is not entirely true. Prior research ([Optimizing Transformer for Low-Resource Neural Machine Translation](https://aclanthology.org/2020.coling-main.304/) and [Revisiting low-resource neural machine translation: A case study](https://arxiv.org/pdf/1905.11901)) show that a lower vocabulary size is better for low-resource NMT, which is the same conclusion as your paper. However, you did not mention anything about their study.\n", "summary_of_the_paper": "The authors explore using a significantly lower vocabulary to improve low-resource NMT. They also experiment with different languages, domains, and architectures.", "summary_of_the_review": "This paper is only comparing 2 approaches: standard subword tokenization vs the proposed quasi-character. The paper is missing several critical comparisons, for example, character-based NMT, and quasi-character on different vocabulary sizes. There is also no report about the performance, despite the author criticizing the performance issue about character-based NMT. Lastly, the paper is also missing related works that have the same claim of low-vocabulary size is better for low resource NMT.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iM6594BlUE", "reviewer": "Reviewer_5PDf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- This paper has a well-defined hypothesis: that quasi-character-level models will outperform 32k-vocabulary-size BPE models at small training set sizes, and tests that hypothesis thoroughly.\n- The experiments on domain adaptation, catastrophic forgetting and vocabulary flexibility are potentially useful for practitioners.\n\nWeaknesses:\n- The work ignores a substantial body of work on character-level translation; there seems to be an abrupt cut-off at 2017. Here is a sampling for missing references that stood out to me: https://arxiv.org/abs/1610.03017;  https://arxiv.org/abs/1808.09943; https://arxiv.org/abs/1810.01480. Likewise, I think it would be useful to cite some of the recent work on character-level large language modeling with Transformers: https://arxiv.org/abs/2103.06874; https://arxiv.org/abs/2105.13626;  https://arxiv.org/abs/2106.12672. Along these lines, I think that it was already known that smaller vocabularies help low-resource languages. For example, Cherry et al (already mentioned; https://arxiv.org/abs/1808.09943), emphasize that character-level models have their greatest advantage when data sizes are small, and Sennrich and Zhang (https://aclanthology.org/P19-1021/) show that reducing vocabulary size improves truly low-resource NMT.\n- The main contribution of this paper seems to be the suggestion that being quasi-character-level allows one to reap the benefits of character-level modeling without paying the computational costs, but this is never demonstrated explicitly. The authors do not try to determine to what degree being quasi-character-level (a) hurts or helps performance with respect to fully character-level; or (b) improves efficiency with respect to fully character-level. A vocabulary-size ablation, graphing speed-versus-quality with a smooth transition of sizes from character to 100 to 400 and then maybe doubling in size thereafter up to 32k would strengthen the work.\n", "summary_of_the_paper": "This paper describes a series of experiments to test the effectiveness of quasi-character-level NMT, which they define as subword vocabularies with sizes between 100 and 400 types. These are tested in the context of artificial low-resource scenarios (i.e.: subsampling medium-sized data sets), and in the context of catastrophic forgetting. In this latter experiment, it is shown that character-level models do suffer from forgetting as they are adapted to other tasks, but their more general vocabulary allows them to be adapted more easily.", "summary_of_the_review": "This paper adds to the ongoing discussion of character-level versus subword-level MT, providing more data points in more architectures indicating the character-level advantage with small training set sizes. It also adds an interesting point about domain flexibility. Overall, the experimental contributions do not seem to be sufficient to warrant a main-conference paper; the results on low-resource scenarios were expected, and the results examining domain flexibility and forgetting are not enough to carry a paper. More experimental emphasis on the utility of quasi-character-level versus character-level, or more experiments on preventing forgetting as suggested in the future work could strengthen the paper substantially.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y3rU5L_b6gw", "reviewer": "Reviewer_7Xbu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I certainly acknowledge the authors’ effort on open vocabulary problems, which is worthy to explore. However, the author merely shows some old news for the MT community, such as character-level NMT works for extremely low resource settings. In addition, many approaches are designed for extremely large vocabulary problems, such as byte-level encodings, e.g. byT5[1], GPT3[2]. Also, with the increase of data volume, extremely low resources are no longer the mainstream scenario for academia and industries. Strategies that work for the rich resources are more likely to be adopted by the community.\n\nThe authors disclose some foundation information, e.g. SELENE project No. 871467 and DeepPattern project No. PROMETRO/2019/121, in acknowledgment, which may violate the double-blind policy.\n\nReference:\n\n[1] Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2021). ByT5: Towards a token-free future with pre-trained byte-to-byte models. ArXiv, abs/2105.13626.\n\n[2] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.J., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.", "summary_of_the_paper": "The authors revisited open vocabulary problems in translation with a character-based approach. They claimed that segmentation approaches such as BPE may be affected by the vocabulary size, which is heuristic. However, purely character-level segmentation makes the sequence too long, thus making the MT model hard to capture long dependencies. \n\nTo achieve a good trade-off, this work explores quasi character level Transformer models for machine translation tasks. They find that the quasi character level MT outperforms their large vocabulary BPE counterparts under the low-resource settings.", "summary_of_the_review": "Reject", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T0MN7rE6EpO", "reviewer": "Reviewer_FBrF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is motivated by an important problem, that of developing a principled method to choose vocabulary sizes for NMT models.\n\nThe paper is well-written, easy to follow, but also short on technical details for experiment descriptions.\n\nThere are several issues in the experiments that are of immediate concern:\n\n1. The experiments on the low-data regime is done synthetically, i.e. no actual low-resource language pairs were used. Why didn't the authors choose an actual low-resource language pair such as Azerbaijani or Tigrinya? The selected languages all use Latin script only, so the language diversity is also quite low in that regard.\n\n2. The current State of the art NMT systems typically use byte-level-fallback in case of <unk> tokens. This mitigates the effect of all long-tailed tokens degenerating to <unk> even for subword models. Why didn't the authors include this fall-back mechanism in their experiments? Since, the handling of long-tailed tokens is described as one of the primary modeling objectives of this paper, the exclusion of this standard technique seems to be an important omission.\n\n3. The most important concern is that vocabularies of 32K were built using 50K, 100K sentences. This deviates from other works (e.g. https://arxiv.org/pdf/2010.04924.pdf, which studies long-tail in NMT reports 10K vocabulary for training data sizes ranging from 10K to 200K and https://arxiv.org/pdf/2004.02334.pdf, which studies subword vocabulary size shows that larger vocabulary size significantly impacts translation under low-resource settings). What this suggests is that 32K might represent a sub-optimal vocabulary size for 50K and 100K training sentences, and given that this hyperparameter is hugely important for establishing the results, the downstream conclusions can't be trusted unambiguously under this setting. Effectively, by training a 32K vocabulary size on 50K sentence pairs, the vocabulary will degenerate to word-level, rendering subword to quasi-character level comparison flawed and inconclusive. Looking at Figure 1, this looks to be the case actually.\n\n4. Certain claims made by the authors aren't quantitatively qualified vey well. For example, \"Quasi-character level transformers achieve better consistency across domains...\" isn't quantified: the absolute BLEU values can't be used to make such claims (they are meaningful for system comparisons only), and consistency isn't explicitly measured.\n\n5. Since this is a MT-focused paper, the authors should also try to include comparisons using state of the art neural metrics such as COMET and BERTScore, rather than using surface-level overlap metrics such as BLEU or ChrF.", "summary_of_the_paper": "This paper inspects the behavior of quasi-character level NMT models, where \"quasi-character level\" is meant to depict subword vocabularies which are an order of magnitude smaller than typical subword vocabulary sizes (e.g. 32K, 64K) used to train state of the art NMT models.\n\nThe dimensions of inspection are:\n\n1. General NMT performance measured by BLEU\n2. Impact of Vocabulary size on fine-tuning performance\n3. Performance comparisons as a function of training data size\n4. Performance measured across different domains and NMT architectures\n\nThe experiments show that smaller subword vocabularies could lead to better performance when training data size is small. Another result is that catastrophic forgetting is more prominent when finetuning models with smaller vocabulary sizes. However, there are several issues in the experimental protocol used, which make the presented conclusions in the paper non-definitive.", "summary_of_the_review": "The paper studies an important problem, but the experimental protocols adopted do not allow any conclusive results. The main claim on the benefit of quasi-character level vocabulary in low-data regime (over a well-selected subword vocabulary size) isn't justified by the presented experiments, due to the issues pointed out above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates the hypothesis that a very small word piece vocabulary (giving a "quasi character level" model) outperforms current methods of neural MT in truly low resource scenarios, and provides some auxiliary studies around word piece frequency and domain transfer. It considers LSTM, CNN, and Transformer NMT models. This is useful information for people working in low resource scenarios to know.

The paper got 3 reviews by people with very strong machine translation expertise. There was a general consensus that the paper was insufficiently aware of prior work on this topic and the paper had problems in experiment construction which raised issues about the comprehensiveness of the result. That is, while this paper adopts a more extremely small vocabulary, Sennrich and Zhang (2017) already showed that a much smaller subword vocabulary can give much stronger results for low resource MT (while Araabi and Monz questioned whether this was as true for Transformer NMT. Meanwhile Cherry et al. (2018) and Kreutzer and Sokolov (2018) argued already the benefits of (almost) character-level NMT. On the experimental side, both not having results on genuinely low-resource scenarios and the commented of Reviewer FBrF that the problem with larger subword vocals here may be mainly due to the small corpus size used for constructing the subword vocabulary are both quite important. Moreover, as mainly an MT experimental study, this paper seems better suited to a more specialized audience of MT researchers at an ACL, WMT, AMTA, etc. venue.

I recommend rejecting this paper as not sufficiently novel, with experiments that need further work, and lacking strong interest to a broader representation learning audience.
Decision: Reject