On the Effectiveness of Quasi Character-Level Models for Machine Translation

1 INTRODUCTION . Neural machine translation ( NMT ) has become the dominant paradigm in the field of machine translation due to the impressive results achieved with encoder-decoder architectures ( Sutskever et al . ( 2014 ) ; Cho et al . ( 2014a ) ; Wu et al . ( 2016 ) ; Vaswani et al . ( 2017 ) ) . Despite these advances , the translation of rare or unknown words has turned out to be a more difficult problem to tackle than was initially thought . For which reason , authors have proposed different approaches to deal with this problem , which can be grouped into three categories : i ) Models based on character-level representations . ii ) Hybrid NMT Models . iii ) Models based on word segmentation . Character-based models are the direct approach as they can naturally deal with rare or unseen words . However , these models have historically resulted in unsatisfactory results ( Vilar et al . ( 2007 ) ; Neubig et al . ( 2013 ) ) or highly expensive models in computational terms ( Luong & Manning ( 2016b ) ) . To balance the benefits of word- and character-based models , Hybrid NMT models were invented ( Luong & Manning ( 2016a ) ) . The idea behind these models is to translate mainly at the word level and only query character components for rare words when necessary . However , these models tend to be a bit cumbersome as they generally need two models to do the back-off . Finally , word segmentation approaches such as BPE ( Sennrich et al . ( 2016 ) ) , Unigram ( Kudo ( 2018 ) ) , or SentencePiece ( Kudo & Richardson ( 2018 ) ) , emerged to address many of the problems mentioned above , by allowing the encoding of rare words using a fixed-vocabulary of subwords . Despite the success of word segmentation models , it is still not clear how to determine the vocabulary size . The most common strategy is to choose an arbitrarily large value ( i.e . 32k ) , under the premise that large vocabularies tend to work well ( Gowda & May ( 2020 ) ) . Despite this heuristic , we think that there must be a better way of choosing this value depending on the dataset properties . Motivated by these ideas , we decided to study whether NMT character-based models had any advantages compared to models with large subword vocabularies ( i.e 32k tokens ) . We theorize that since words follow a Zipfian distribution , models with large vocabularies would require large datasets to learn the long-tailed distribution of subwords . In contrast , character-based models would be able to learn it more easily in data-poor environments as their subword distribution is less pronounced than the distribution for large vocabularies . One of the main criticisms of character-level models is that they involve longer sequences and therefore , more long-term dependencies . To alleviate some of these problems , we decided to study quasi character-level models , which are essentially small subword-level vocabularies ( i.e 100-400 tokens ) . Furthermore , we decided to study if character-based models can mitigate the effects of catastrophic forgetting based on the idea that , if this phenomenon depends on the vocabulary ’ s domain , when a vocabulary is shared between domains , these effects should be mitigated . The contributions of this paper are fourfold : • Quasi character-level Transformers seem to consistently outperform large subword-level Transformers on low-resource environments . ( Section 5.1.1 ) . • When LSTMs and CNNs are used , this phenomenon is still present but it is more sensitive on the chosen hyperparameters . ( Section 5.1.3 ) . • Quasi character-level Transformers appear to be more susceptible to catastrophic forgetting than their large subword-level counterparts . ( Section 5.2 ) . • However , quasi character-level Transformers achieved better cross-domain consistencies . ( Section 5.2 ) . 2 RELATED WORK . Character-based models have been well-studied in the Natural Language Processing ( NLP ) field to deal with the open-vocabulary problem . One of the first character-based models was proposed by Vilar et al . ( 2007 ) , who treated the source and target sentences as a string of letters . Similarly , Neubig et al . ( 2013 ) viewed translation as a single transduction between character strings in the source . However , despite the relevance of their contributions , their results were not satisfactory as their models generally performed worse than their word-based counterparts . Consequently , authors began to propose strategies to get the best of word- and character-level models . These approaches were mostly based on Hybrid NMT models or subword-level models with vocabularies built through word segmentation techniques . Luong & Manning ( 2016a ) achieved open vocabulary NMT through hybrid models , which translate at the word level and consult the character components only for rare words ; and Sennrich et al . ( 2016 ) developed a model to segment words based on the Byte-Pair encoding compression algorithm proposed by Gage ( 1994 ) . Later , more word segmentation models were proposed based on this idea , such as Unigram by Kudo ( 2018 ) or SentencePiece by Kudo & Richardson ( 2018 ) . Chung et al . ( 2016 ) demonstrated that a NMT model with a character-based decoder can outperform NMT models with subword-level decoders . Similarly , Luong & Manning ( 2016b ) and Costa-jussà & Fonollosa ( 2016 ) showed that competitive purely character-based NMT models were possible . Despite their relative success , these models were extremely slow to train and at runtime . Recurrent neural architectures have been the dominant neural architecture for NLP tasks ( Cho et al . ( 2014b ) ; Wu et al . ( 2016 ) ; Sundermeyer et al . ( 2014 ) ) . However , in recent years we have seen convolutional neural models outperform recurrent models on some NLP tasks . Gehring et al . ( 2017 ) proposed a competitive sequence to sequence architecture based entirely on convolutional neural networks , and Conneau et al . ( 2016 ) achieved state-of-the-art machine translation with a purely character-level NMT model via deep convolutional stacks . Recently , the attention mechanism ( Bahdanau et al . ( 2015 ) ; Luong et al . ( 2015 ) ) along with the Transformer architecture by Vaswani et al . ( 2017 ) , has led to significant progress in the field of NLP , due to the parallelizable nature of the Transformer architectures and its ability to deal with long-term dependencies . This paper ends with a brief discussion on the ability of quasi character-based Transformers to mitigate the catastrophic forgetting problem in machine translation . As far as we know , this is the first work to address this issue from the perspective of ( quasi ) character-level models , since most of the existing alternatives are based on regularization ( i.e . LwF by Li & Hoiem ( 2016 ) , EWC by Kirkpatrick et al . ( 2016 ) ) , dynamic architectures ( Rusu et al . ( 2016 ) ; Draelos et al . ( 2016 ) ) or Complementary Learning Systems ( CLS ) and Memory Replay ( MR ) ( Kemker & Kanan ( 2017 ) ; Lopez-Paz & Ranzato ( 2017 ) ) . 3 NEURAL MACHINE TRANSLATION . 3.1 NEURAL ARCHITECTURES FOR MACHINE TRANSLATION . The goal of any translation system is to transform an input sequence in a given language into an output sequence in a target language . Nowadays , this is usually done using neural models based on the encoder-decoder architecture ; also know as seq2seq models in the machine translation community ( Sutskever et al . ( 2014 ) ) . The encoder part transforms the input sequence into an internal representation , and then the decoder transforms this internal representation into the output sequence . In this work , we focus our study on the Transformer architecture , although recurrent and fully convolutional architectures are also explored for completeness . Recurrent architectures ( RNNs ) were the first to be successfully applied in an encoder-decoder setup for machine translation . Even though there are many types of RNNs , most of them chain sequentially a series of unit cells to process temporal sequences . We decided to use LSTMs ( Hochreiter & Schmidhuber ( 1997 ) ) because their units cells are explicitly designed to deal with long-term dependencies . Convolution-based architectures ( CNN ) do not contain any recurrent elements . They can do this , because the idea behind this architecture is that the convolutional filters can slide through the sequence of tokens , from beginning to end ( Gehring et al . ( 2017 ) ) . Initially , the convolutional layers were intended to operate on 2-dimensional images so , in order to apply them to text data , we have to imagine the sequence of tokens as if it were a 1xm image , where the height of the filter is equal to 1 , and the width equal to k. Being k the number of consecutive tokens that the filter can observe at a given time . Finally , we have Transformer architecture by Vaswani et al . ( 2017 ) , a state-of-the-art architecture based entirely on the concept of attention ( Bahdanau et al . ( 2015 ) ; Luong et al . ( 2015 ) ) to draw global dependencies between the input and output . This architecture does not use any recurrence or convolutional layer to process temporal sequences . Instead , it makes use of linear and normalization layers , residual connections , attention mechanisms , and some tricks to encode temporal information . 3.2 THE OPEN VOCABULARY PROBLEM . In the written language it is common to find alternative spellings ( i.e . color-colour ) and typos ( i.e acknowledge-acknowlege ) that slightly modify the spelling of a word but do not prevent us , the humans , from knowing its meaning . However , if a model is using a word-level representation , it will stop knowing a known word at the very first moment that it is slightly modified and this modification is not in its vocabulary . Furthermore , it must be taken into account that most languages make use of agglutination and compounding mechanisms to form new words , which in itself makes it very inefficient to build a vocabulary with all possible words of a language . Because of this , researchers have proposed multiple approaches to deal with the open vocabulary problem . These approaches can be mostly grouped into three categories : i ) Character-based models ii ) Hybrid models iii ) Subword-based models . Character-based models contain the minimum possible vocabulary with which to form all possible words . Consequently , these models can translate rare or even unseen words character-by-character , but at the same time , these models tend to be much slower and harder to train than word-based models . This is because they have to deal with longer sequences and therefore , longer long-term dependencies . Hybrid models tend to translate primarily at word-level but fall back to character-level translation when a rare or unseen word appears . Finally , subword-based models are fixed close-vocabularies that use subwords instead of full words . These models allow us to represent infrequent or even unknown words as a sequence of subwords , but without the performance and memory issues that typically imply character-based models . In this work , we always work at the subword level but distinguishing two types of subword vocabularies . The first one is the quasi character-level vocabulary with 100-400 tokens , and the second one is a large subword-level vocabulary with around 32,000 tokens . To build our subword-level vocabularies we chose a word segmentation model known as Byte-Pair Encoding ( BPE ) by Sennrich et al . ( 2016 ) . We did also consider Unigram ( Kudo ( 2018 ) ) and SentencePiece ( Kudo & Richardson ( 2018 ) ) but eventually , we decided to use BPE due to its popularity . The idea behind BPE is to start with a set of characters and iteratively extend it with the most frequent n-grams pairs in the corpus .

Summary: This paper inspects the behavior of quasi-character level NMT models, where "quasi-character level" is meant to depict subword vocabularies which are an order of magnitude smaller than typical subword vocabulary sizes (e.g. 32K, 64K) used to train state of the art NMT models.  The dimensions of inspection are:  1. General NMT performance measured by BLEU 2. Impact of Vocabulary size on fine-tuning performance 3. Performance comparisons as a function of training data size 4. Performance measured across different domains and NMT architectures  The experiments show that smaller subword vocabularies could lead to better performance when training data size is small. Another result is that catastrophic forgetting is more prominent when finetuning models with smaller vocabulary sizes. However, there are several issues in the experimental protocol used, which make the presented conclusions in the paper non-definitive.