How BPE Affects Memorization in Transformers

Training data memorization in NLP can both be beneficial ( e.g. , closed-book QA ) and undesirable ( personal data extraction ) . In any case , successful model training requires a non-trivial amount of memorization to store word spellings , various linguistic idiosyncrasies and common knowledge . However , little is known about what affects the memorization behavior of NLP models , as the field tends to focus on the equally important question of generalization . In this work , we demonstrate that the size of the subword vocabulary learned by Byte-Pair Encoding ( BPE ) greatly affects both ability and tendency of standard Transformer models to memorize training data , even when we control for the number of learned parameters . We find that with a large subword vocabulary size , Transformer models fit random mappings more easily and are more vulnerable to membership inference attacks . Similarly , given a prompt , Transformer-based language models with large subword vocabularies reproduce the training data more often . We conjecture this effect is caused by reduction in the sequences ’ length that happens as the BPE vocabulary grows . Our findings can allow a more informed choice of hyper-parameters , that is better tailored for a particular use-case . 1 INTRODUCTION . The Transformer architecture ( Vaswani et al. , 2017 ) became the backbone of the state-of-the-art models in a variety of tasks ( Liu et al. , 2019 ; Raffel et al. , 2019 ; Adiwardana et al. , 2020 ; Brown et al. , 2020 ) . This spurred a significant interest in better understanding inner workings of these models ( Vig and Belinkov , 2019 ; Clark et al. , 2019 ; Kharitonov and Chaabouni , 2020 ; Hahn , 2020 ; Movva and Zhao , 2020 ; Chaabouni et al. , 2021 ; Merrill et al. , 2021 ; Sinha et al. , 2021 ) . Most of these works have focussed specifically on how models generalize and capture structure across samples that are similar . For instance , Vig and Belinkov ( 2019 ) focussed on how attention align with specific syntactic dependency relations , Hupkes et al . ( 2020 ) considered if Transformers generalize compositionally and Kharitonov and Chaabouni ( 2020 ) studied how different models generalize from very few data . In contrast to these studies , we focus on factors that control the training data memorization behavior of Transformers , which we believe to be important for several reasons . First , large Transformer models are increasingly often used as a storage , for instance , as a generalpurpose knowledge base or as a closed-book question-answering system ( Petroni et al. , 2019 ; Roberts et al. , 2020 ; Lewis et al. , 2020 ) . Clearly , the ability to memorize factual knowledge from the training data is crucial for such applications . There are even Transformers models that are explicitly endowed with an external training data memorization mechanism ( Khandelwal et al. , 2020 ; 2021 ; He et al. , 2021 ) , demonstrating that further boosting their memorization abilities is beneficial . Second , in contrast , the same ability to memorize can become undesirable and lead to a leakage of personal data from trained models ( Carlini et al. , 2020 ; Thakkar et al. , 2021 ) . A better understanding of the phenomenon is thus instrumental both to enable better memorization when it is needed and to avoid it when not . Third , while generalization and memorization are often thought of as competing modes of fitting data , training effective models in real tasks requires a non-trivial combination of the two . For instance , successful language models need to generalize to be able to deal with never-seen-before sentences , but they also need to memorize the spelling of words , the non-compositional meaning of idioms , idiosyncrasies of languages , common knowledge , etc ( see , e.g . Dankers et al. , 2021 ) .1 Despite this apparent importance , there is very little research into memorization in Transformers and in NLP models in general , and we have only superficial understanding of what factors affect this behavior . Intuitively , the number of parameters , data augmentation , and regularization are likely to affect how successful models are in memorization ( Zhang et al. , 2016 ; Sablayrolles et al. , 2018 ) . In this work , we primarily focus on the influence of a less obvious yet important factor : we study how the selection of modelling units affects memorization . Typically , the same data can be represented on various levels : raw bytes and their groups , individual characters , subword units , and entire words . A very common approach is to learn subword-level vocabulary with Byte-Pair Encoding ( BPE ) ( Sennrich et al. , 2015 ) or similar methods ( e.g. , Devlin et al. , 2018 ; Kudo and Richardson , 2018 ; Provilkov et al. , 2019 ) . In spite of ubiquitous use of these methods , to the best of our knowledge , there is no clear understanding of how the number of subwords or BPE operations should be chosen and how this affect behavior of a model . We expect that BPE-like segmentation might play a crucial role in memorization , as it controls the trade-off between the number of primitives a model will have to operate with and the lengths of sequences it must represent . In this work , to characterize a model ’ s behavior , we measure three “ facets ” of training data memorization . First , as a proxy for the memorization capacity of a model , we use its ability to fit random , non-systematic mappings . Next , we study the preference for memorization when generalization is possible . For that , we study how easy it is to accurately tell if a particular example was used in the model ’ s training data via a membership inference attack ( Shokri et al. , 2017 ) . Finally , we examine how easy it is to recover training data from a trained language model . We experiment with three Transformer architectures : causal & masked language models , and encoder-based classifiers . Our main experimental finding is that , across all architectures and tasks , the choice of modeling units strongly affects the memorization behavior of the models , with large-cardinality BPE vocabularies greatly facilitating memorization . This observation holds even when we control for the number of trainable parameters . After establishing this fact , we look deeper into the causes of the phenomenon we observe . We examine three candidate causes which are principal ( side- ) effects of applying BPE : ( i ) removing redundancy in the data ( due to compression ) , ( ii ) increase in the number of the unique units used to represent the data , or ( iii ) reducing the length of the training sequences . By finding a similar effect with incompressible randomly generated data we can rule out the first possibility . Next , we artificially double the vocabulary size by introducing “ synonym ” tokens and observe that the vocabulary growth , in isolation , leads to a different memorization pattern . Thus , by exclusion , we conjecture that reducing utterance length is , at least , a very important factor of memorization.2 . 2 STUDYING MEMORIZATION – THE TASKS . To quantify the memorization capabilities and preferences of NLP models , we use three different setups , with which we aim to cover different facets of what one can call training data memorization . 2.1 LEARNING MAPPINGS WITH RANDOM LABELS . Firstly , we consider a task of learning non-systematic mappings , where labels are independent from inputs ( Zhang et al. , 2016 ) . To achieve accuracy above chance , the model has to “ store ” the training example in some way , thus we assume that a higher training accuracy implies increased training data memorization ability . To experiment with realistic natural input data , we consider the Stanford Natural Language Inference dataset ( SNLI , Bowman et al. , 2015 ) . In this dataset , each example is a pair of two sentences , one representing a premise ( “ A boy is jumping on skateboard in the middle of a red bridge. ” ) and the other representing a hypothesis ( “ The boy does a skateboarding trick. ” ) . Each example is assigned a 1An interesting example is the language model of Lakhotia et al . ( 2021 ) , which is trained on sub-phonemic acoustic units without word boundaries , but confidently processes a large vocabulary of English words . 2Another potential cause that we consider is the changes in the relative frequencies of tokens that BPE brings along . In Appendix D we investigate and rule out this hypothesis . label that denotes if the hypothesis entails the premise , contradicts it , or neither contradict nor entails ( neutral ) . We represent examples in a concatenated form with a separator token ( : : ) between a premise and a hypothesis ( “ A boy is jumping on skateboard in the middle of a red bridge . : : The boy does a skateboarding trick. ” ) . For uniformity with other experiments , we transform the dataset into a binary classification task by filtering out all examples with neutral labels.3 After this filtering , 367,388 examples remain . We replace original labels with randomly sampled ones ( -1 / +1 , equiprobably ) , and we measure memorization by measuring the ( training ) accuracy of the models on this data . 2.2 MEMBERSHIP INFERENCE . While the task of memorizing random labels can tell us how much a model can remember , it doesn ’ t allow us to test how much or what a model memorizes when it is trained on tasks which also admit ( or require ) generalization . As this is typically the case with natural data , we thus need a different strategy to assess memorization in more realistic scenarios . To evaluate memorization in such cases , we resort to measuring models ’ vulnerability to membership inference attacks ( Shokri et al. , 2017 ) . Indeed , if it is “ easy ” to accurately tell if a particular example was used for training , we assume it is actually “ stored ” in the weights of the model in some form , rather than being inferred from a more general rule or pattern ( which would lead to high scores also for examples that were not in the training data , but that are likely given that data ) . More formally , suppose we have a model fθ that was trained on a subset D′ of a large set of examples D , ( D′ ⊂ D ) withD′ obtained by sampling examples independently fromD with some probability λ . The goal of membership inference is to figure out , given fθ , whether a particular example ( xi , yi ) ∈ D was included in the training data D′ . We implement a simple membership inference attack protocol by Yeom et al . ( 2018 ) . Given a model fθ parameterized by θ , we calculate its loss on a data point l ( fθ ( xi ) , yi ) and compare to a threshold τ : if it is below the threshold , the data point belongs to the training data . By controlling τ we can control the trade-off between precision and recall of the attack . To avoid the dependency on this parameter and to represent the entire space of possible trade-offs , we use the AUC metric . After training a model , we measure the AUC of the above rule that separates training and hold-out examples . In this set of experiments , we again use the SNLI dataset . However , in this experiment we use the true labels of the dataset , rather than using the random labels of the previous setup , allowing us to consider both generalization and memorization . To make the prior probability of an example belonging to the training dataset equal to 12 , at training time we use only a half of the original ’ s dataset training data ( 367,388 examples remaining after filtering ) , with the second half playing the role of the hold-out .

Summary: This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.