On the Effectiveness of Quasi Character-Level Models for Machine Translation

1 INTRODUCTION . Neural machine translation ( NMT ) has become the dominant paradigm in the field of machine translation due to the impressive results achieved with encoder-decoder architectures ( Sutskever et al . ( 2014 ) ; Cho et al . ( 2014a ) ; Wu et al . ( 2016 ) ; Vaswani et al . ( 2017 ) ) . Despite these advances , the translation of rare or unknown words has turned out to be a more difficult problem to tackle than was initially thought . For which reason , authors have proposed different approaches to deal with this problem , which can be grouped into three categories : i ) Models based on character-level representations . ii ) Hybrid NMT Models . iii ) Models based on word segmentation . Character-based models are the direct approach as they can naturally deal with rare or unseen words . However , these models have historically resulted in unsatisfactory results ( Vilar et al . ( 2007 ) ; Neubig et al . ( 2013 ) ) or highly expensive models in computational terms ( Luong & Manning ( 2016b ) ) . To balance the benefits of word- and character-based models , Hybrid NMT models were invented ( Luong & Manning ( 2016a ) ) . The idea behind these models is to translate mainly at the word level and only query character components for rare words when necessary . However , these models tend to be a bit cumbersome as they generally need two models to do the back-off . Finally , word segmentation approaches such as BPE ( Sennrich et al . ( 2016 ) ) , Unigram ( Kudo ( 2018 ) ) , or SentencePiece ( Kudo & Richardson ( 2018 ) ) , emerged to address many of the problems mentioned above , by allowing the encoding of rare words using a fixed-vocabulary of subwords . Despite the success of word segmentation models , it is still not clear how to determine the vocabulary size . The most common strategy is to choose an arbitrarily large value ( i.e . 32k ) , under the premise that large vocabularies tend to work well ( Gowda & May ( 2020 ) ) . Despite this heuristic , we think that there must be a better way of choosing this value depending on the dataset properties . Motivated by these ideas , we decided to study whether NMT character-based models had any advantages compared to models with large subword vocabularies ( i.e 32k tokens ) . We theorize that since words follow a Zipfian distribution , models with large vocabularies would require large datasets to learn the long-tailed distribution of subwords . In contrast , character-based models would be able to learn it more easily in data-poor environments as their subword distribution is less pronounced than the distribution for large vocabularies . One of the main criticisms of character-level models is that they involve longer sequences and therefore , more long-term dependencies . To alleviate some of these problems , we decided to study quasi character-level models , which are essentially small subword-level vocabularies ( i.e 100-400 tokens ) . Furthermore , we decided to study if character-based models can mitigate the effects of catastrophic forgetting based on the idea that , if this phenomenon depends on the vocabulary ’ s domain , when a vocabulary is shared between domains , these effects should be mitigated . The contributions of this paper are fourfold : • Quasi character-level Transformers seem to consistently outperform large subword-level Transformers on low-resource environments . ( Section 5.1.1 ) . • When LSTMs and CNNs are used , this phenomenon is still present but it is more sensitive on the chosen hyperparameters . ( Section 5.1.3 ) . • Quasi character-level Transformers appear to be more susceptible to catastrophic forgetting than their large subword-level counterparts . ( Section 5.2 ) . • However , quasi character-level Transformers achieved better cross-domain consistencies . ( Section 5.2 ) . 2 RELATED WORK . Character-based models have been well-studied in the Natural Language Processing ( NLP ) field to deal with the open-vocabulary problem . One of the first character-based models was proposed by Vilar et al . ( 2007 ) , who treated the source and target sentences as a string of letters . Similarly , Neubig et al . ( 2013 ) viewed translation as a single transduction between character strings in the source . However , despite the relevance of their contributions , their results were not satisfactory as their models generally performed worse than their word-based counterparts . Consequently , authors began to propose strategies to get the best of word- and character-level models . These approaches were mostly based on Hybrid NMT models or subword-level models with vocabularies built through word segmentation techniques . Luong & Manning ( 2016a ) achieved open vocabulary NMT through hybrid models , which translate at the word level and consult the character components only for rare words ; and Sennrich et al . ( 2016 ) developed a model to segment words based on the Byte-Pair encoding compression algorithm proposed by Gage ( 1994 ) . Later , more word segmentation models were proposed based on this idea , such as Unigram by Kudo ( 2018 ) or SentencePiece by Kudo & Richardson ( 2018 ) . Chung et al . ( 2016 ) demonstrated that a NMT model with a character-based decoder can outperform NMT models with subword-level decoders . Similarly , Luong & Manning ( 2016b ) and Costa-jussà & Fonollosa ( 2016 ) showed that competitive purely character-based NMT models were possible . Despite their relative success , these models were extremely slow to train and at runtime . Recurrent neural architectures have been the dominant neural architecture for NLP tasks ( Cho et al . ( 2014b ) ; Wu et al . ( 2016 ) ; Sundermeyer et al . ( 2014 ) ) . However , in recent years we have seen convolutional neural models outperform recurrent models on some NLP tasks . Gehring et al . ( 2017 ) proposed a competitive sequence to sequence architecture based entirely on convolutional neural networks , and Conneau et al . ( 2016 ) achieved state-of-the-art machine translation with a purely character-level NMT model via deep convolutional stacks . Recently , the attention mechanism ( Bahdanau et al . ( 2015 ) ; Luong et al . ( 2015 ) ) along with the Transformer architecture by Vaswani et al . ( 2017 ) , has led to significant progress in the field of NLP , due to the parallelizable nature of the Transformer architectures and its ability to deal with long-term dependencies . This paper ends with a brief discussion on the ability of quasi character-based Transformers to mitigate the catastrophic forgetting problem in machine translation . As far as we know , this is the first work to address this issue from the perspective of ( quasi ) character-level models , since most of the existing alternatives are based on regularization ( i.e . LwF by Li & Hoiem ( 2016 ) , EWC by Kirkpatrick et al . ( 2016 ) ) , dynamic architectures ( Rusu et al . ( 2016 ) ; Draelos et al . ( 2016 ) ) or Complementary Learning Systems ( CLS ) and Memory Replay ( MR ) ( Kemker & Kanan ( 2017 ) ; Lopez-Paz & Ranzato ( 2017 ) ) . 3 NEURAL MACHINE TRANSLATION . 3.1 NEURAL ARCHITECTURES FOR MACHINE TRANSLATION . The goal of any translation system is to transform an input sequence in a given language into an output sequence in a target language . Nowadays , this is usually done using neural models based on the encoder-decoder architecture ; also know as seq2seq models in the machine translation community ( Sutskever et al . ( 2014 ) ) . The encoder part transforms the input sequence into an internal representation , and then the decoder transforms this internal representation into the output sequence . In this work , we focus our study on the Transformer architecture , although recurrent and fully convolutional architectures are also explored for completeness . Recurrent architectures ( RNNs ) were the first to be successfully applied in an encoder-decoder setup for machine translation . Even though there are many types of RNNs , most of them chain sequentially a series of unit cells to process temporal sequences . We decided to use LSTMs ( Hochreiter & Schmidhuber ( 1997 ) ) because their units cells are explicitly designed to deal with long-term dependencies . Convolution-based architectures ( CNN ) do not contain any recurrent elements . They can do this , because the idea behind this architecture is that the convolutional filters can slide through the sequence of tokens , from beginning to end ( Gehring et al . ( 2017 ) ) . Initially , the convolutional layers were intended to operate on 2-dimensional images so , in order to apply them to text data , we have to imagine the sequence of tokens as if it were a 1xm image , where the height of the filter is equal to 1 , and the width equal to k. Being k the number of consecutive tokens that the filter can observe at a given time . Finally , we have Transformer architecture by Vaswani et al . ( 2017 ) , a state-of-the-art architecture based entirely on the concept of attention ( Bahdanau et al . ( 2015 ) ; Luong et al . ( 2015 ) ) to draw global dependencies between the input and output . This architecture does not use any recurrence or convolutional layer to process temporal sequences . Instead , it makes use of linear and normalization layers , residual connections , attention mechanisms , and some tricks to encode temporal information . 3.2 THE OPEN VOCABULARY PROBLEM . In the written language it is common to find alternative spellings ( i.e . color-colour ) and typos ( i.e acknowledge-acknowlege ) that slightly modify the spelling of a word but do not prevent us , the humans , from knowing its meaning . However , if a model is using a word-level representation , it will stop knowing a known word at the very first moment that it is slightly modified and this modification is not in its vocabulary . Furthermore , it must be taken into account that most languages make use of agglutination and compounding mechanisms to form new words , which in itself makes it very inefficient to build a vocabulary with all possible words of a language . Because of this , researchers have proposed multiple approaches to deal with the open vocabulary problem . These approaches can be mostly grouped into three categories : i ) Character-based models ii ) Hybrid models iii ) Subword-based models . Character-based models contain the minimum possible vocabulary with which to form all possible words . Consequently , these models can translate rare or even unseen words character-by-character , but at the same time , these models tend to be much slower and harder to train than word-based models . This is because they have to deal with longer sequences and therefore , longer long-term dependencies . Hybrid models tend to translate primarily at word-level but fall back to character-level translation when a rare or unseen word appears . Finally , subword-based models are fixed close-vocabularies that use subwords instead of full words . These models allow us to represent infrequent or even unknown words as a sequence of subwords , but without the performance and memory issues that typically imply character-based models . In this work , we always work at the subword level but distinguishing two types of subword vocabularies . The first one is the quasi character-level vocabulary with 100-400 tokens , and the second one is a large subword-level vocabulary with around 32,000 tokens . To build our subword-level vocabularies we chose a word segmentation model known as Byte-Pair Encoding ( BPE ) by Sennrich et al . ( 2016 ) . We did also consider Unigram ( Kudo ( 2018 ) ) and SentencePiece ( Kudo & Richardson ( 2018 ) ) but eventually , we decided to use BPE due to its popularity . The idea behind BPE is to start with a set of characters and iteratively extend it with the most frequent n-grams pairs in the corpus .

Summary: This paper describes a series of experiments to test the effectiveness of quasi-character-level NMT, which they define as subword vocabularies with sizes between 100 and 400 types. These are tested in the context of artificial low-resource scenarios (i.e.: subsampling medium-sized data sets), and in the context of catastrophic forgetting. In this latter experiment, it is shown that character-level models do suffer from forgetting as they are adapted to other tasks, but their more general vocabulary allows them to be adapted more easily.