Title: Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions
Venue: NeurIPS 2024
Accepted: yes
Keywords: tokenizers

distribution inference

security
Abstract: The pretraining data of today's strongest language models remains opaque, even when their parameters are open-sourced.
In particular, little is known about the proportions of different domains, languages, or code represented in the data. While a long line of membership inference attacks aim to identify training examples on an instance level, they do not extend easily to *global* statistics about the corpus. In this work, we tackle a task which we call *data mixture inference*, which aims to uncover the distributional make-up of the pretraining data. We introduce a novel attack based on a previously overlooked source of information — byte-pair encoding (BPE) tokenizers, used by the vast majority of modern language models. Our key insight is that the ordered vocabulary learned by a BPE tokenizer naturally reveals information about the token frequencies in its training data: the first token is the most common byte pair, the second is the most common pair after merging the first token, and so on. Given a tokenizer's merge list along with data samples for each category of interest (e.g., different natural languages), we formulate a linear program that solves for the relative proportion of each category in the tokenizer's training set. Importantly, to the extent to which tokenizer training data is representative of the pretraining data, we indirectly learn about the pretraining data. In controlled experiments, we show that our attack can recover mixture ratios with high precision for tokenizers trained on known mixtures of natural languages, programming languages, and data sources. We then apply our approach to off-the-shelf tokenizers released alongside recent LMs. We confirm much publicly disclosed information about these models, and also make several new inferences: GPT-4o is much more multilingual than its predecessors, training on 10x more non-English data than GPT-3.5, Llama 3 and Claude are trained on predominantly code, and many recent models are trained on 7-16% books. We hope our work sheds light on current design practices for pretraining data, and inspires continued research into data mixture inference for LMs.

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

Abstract

The pretraining data of today's strongest language models is opaque; in particular, little is known about the proportions of various domains or languages represented. In this work, we tackle a task which we call data mixture inference , which aims to uncover the distributional make-up of training data. We introduce a novel attack based on a previously overlooked source of information: byte-pair encoding (BPE) tokenizers, used by the vast majority of modern language models. Our key insight is that the ordered list of merge rules learned by a BPE tokenizer naturally reveals information about the token frequencies in its training data. Given a tokenizer's merge list along with example data for each category of interest, we formulate a linear program that solves for the proportion of each category in the tokenizer's training set. In controlled experiments, we show that our attack recovers mixture ratios with high precision for tokenizers trained on known mixtures of natural languages, programming languages, and data sources. We then apply our approach to off-the-shelf tokenizers released with recent LMs. We confirm much publicly disclosed information about these models, and also make several new inferences: GPT-4O and MISTRAL NEMO's tokenizers are much more multilingual than their predecessors, training on 39% and 47% non-English language data, respectively; LLAMA 3 extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT3.5's and CLAUDE's tokenizers are trained on predominantly code ( ∼ 60 %). We hope our work sheds light on current design practices for pretraining data, and inspires continued research into data mixture inference for LMs. 1

1 Introduction

Pretraining data is at the heart of language model development, yet it remains a trade secret for today's strongest models. While it has become more common for model-producing organizations to release model parameters, they rarely share the pretraining data or important details about its construction. In particular, little is known about the proportion of different languages, code, or data sources present in the data; these design decisions require extensive experimentation that few organizations have the resources to perform, and have a significant impact on the resulting LM [7, 39, 35, 38, 49, 59].

While a long line of membership inference attacks [15, 51, 43, 52, 16, 21] aim to reveal information about the model's pretraining data, they typically focus on testing whether particular instances or authors contributed to the data. In this work, we tackle a different task we call data mixture inference which, given a set of disjoint categories that cover the training data (e.g., the set of natural and programming languages), aims to uncover each of their proportions.

♡♣

English

DEn

Normalize the digits, then

x = logits.softmax()

# get probs

ensure that they sum to 1.

mixture ratios

DPy

DPy

DEn

DEn

R

Given data, BPE

1

t

1

(

)

Given ordered merge

learns an ordered

list of merge rules

2

_t h

2

i

t

rules, can we solve for

3

_th

e

3

t

the mixture ratios?

4

S

u

4

S

u

5

it

5

#

6

（)

6

The learned merge list is sensitive to the mixture ratios!

To this end, we identify a previously overlooked source of information: trained byte-pair encoding tokenizers (BPE; [50]), which are the near-universal choice for modern language models. Our key insight is that the ordered merge rules learned by a BPE tokenizer naturally reveal information about the frequency of tokens in the tokenizer's training data. During training, the BPE algorithm iteratively finds the ordered pair of tokens with the highest frequency, adds it to the merge list, and applies the merge to the dataset. Therefore, if the pair ( ; , \n ) was merged in the 52 nd step (as is the case for GPT-4O), then it must be the most frequent pair in the data after applying the preceding 51 merges; in this case, it is a signature of substantial code data. Note that at inference-time, new text is tokenized by applying the learned merge rules in-order. Open-source models require open tokenizers; even closed models often have open tokenizers for the purpose of estimating query cost.

Our method builds a linear program where the constraints are derived from the true most-frequent merge at every step in the merge list, and solves for the proportions of each category. We first demonstrate its effectiveness in controlled experiments where we train tokenizers on known mixtures of data. We consider three kinds of data mixtures: natural languages, programming languages, and data sources. Our method is highly effective, achieving accuracy between two and five orders of magnitude better than baselines based on tokenizer efficiency or inspection of the vocabulary.

Then, we apply our method to infer previously unknown distributional information about off-the-shelf, commercial tokenizers (the top of these merge lists are shown in §C.2 for qualitative inspection). We consider all tokenizers released with GPT, LLAMA, and MISTRAL model families, as well as GPT-NEOX, GEMMA, and CLAUDE, which we will refer to later using their associated model names. We corroborate reported information and public intuition about these tokenizers with exact numbers GPT-2 is trained on predominantly English (99%), GPT-3.5 is the first in the GPT family to be trained extensively on code (63%), and LLAMA trains on only languages that use Latin or Cyrillic scripts. We also make several new inferences: GPT-4O is much more multilingual than its predecessors, training on 39% non-English text, with 68 languages that make up at least 0.1% of the data. LLAMA 3 extends GPT-3.5's tokenizer primarily for multilingual use, using 48% non-English text (and 30% code data). Finally, we surprisingly infer that all tokenizers we study are trained on 7% - 26% book data, potentially because books use a more standard vocabulary compared to the web.

Inferring tokenizer training data mixtures has several important implications. Ideally, the tokenizer training data is representative of the LM's pretraining data [58]; disconnects can lead to poor encoding

English

Code

Languages with

Latin + Cyrillic scripts

Other languages

0.0

0.2

0.4

0.6

0.8

1.0

Proportion

99

1

0

0

34

63

2

1

28

33

18

21

49

19

27

4

21

30

22

27

58

26

14

3

35

18

25

22

82

15

2

1

39

57

2

2

46

26

17

11

GPT-2

GPT-3.5

GPT-4o

Llama

Llama 3*

Mistral

Mistral NeMo

GPT-NeoX

Gemma

of the pretraining text [2] and potential for 'glitch tokens' that trigger degenerate model behavior [48, 30, 33]. Additionally, the tokenizer can be seen as a leading indicator of the model developer's priorities, as it is often designed to accommodate future models. Tokenizer training mixtures may also be used to accelerate model-based attacks, for instance by suggesting data categories to prioritize for membership inference. Finally, it can enable external auditing of training data for biases, by identifying under-represented languages or data sources.

2 Background: BPE tokenizers

Byte-pair encoding (BPE), introduced by Sennrich et al. [50] for NLP, 2 is a tokenization algorithm that learns subword-based encodings from training data. At a high level, the algorithm iteratively merges frequently co-occurring pairs of tokens until the desired vocabulary size is reached.

More precisely, the training text is first pretokenized by splitting it into 'words' that limit the extent of tokenization. Merges cannot bridge these words, and thus the final learned tokens will be parts of these words. Pretokenization can be as simple as splitting on whitespace, so that common sequences of words (e.g., ' it is ') do not become a single token.

After pretokenization, the words are split into bytes, which form the starting vocabulary. Then, the BPE algorithm iteratively counts the frequency of each neighboring pair of tokens and picks the most frequent to be merged next. This merge is added to the merge list and applied to the entire text, and the merged token is added to the vocabulary. For instance, if the merge is ( th , e ), then all instances of the token sequence th , e will be replaced with the , which is added to the vocabulary. BPE then updates the frequencies of all pairs, and identifies the next most frequent. This continues until the desired vocabulary size is reached. At the end of training, the algorithm has learned an ordered list of merge rules m (1) , . . . , m ( M ) .

To tokenize new text, the tokenizer splits the text into bytes and applies the learned merge rules, in order. As we will see, the merge list reflects rich distributional information about the training data.

3 Data mixture inference attack

Suppose we have a set of n data categories of interest, and data distributions {D i } n i =1 for each one. Then suppose we receive a BPE tokenizer, which was trained on a large sample of text from the mixture ∑ n i =1 α ∗ i D i with non-negative weights α ∗ ∈ R n satisfying ∑ n i =1 α ∗ i = 1 . Given corpora

English

Spanish

This is English.

Esto es espanol.

α

α,[１１00 ０ ０] +α.[0 ０ ０１１１ 2] =

1

e

S

En

0

Th hi is Es st to

es

ThhiisEs st to(

es

CEn

2

is

αe,[1１ 000 0]+α,[00０１１１1] =

α

En

ThhiisEssttoesp

Thhiis Es st to esp

CEn

0

{ D i } n i =1 sampled from each of the D i respectively, the goal of data mixture inference is to produce a good estimate ˆ α of α ∗ .

Now we describe how to set up the set of constraints that make up a linear program whose solution is this estimate (§3.1), reduce the storage requirements (§3.2), and improve efficiency (§3.3, §3.4).

3.1 Data mixture inference via linear programming

We build a linear program (LP) with variables α and constraints derived using information from the tokenizer and our sample corpora. The given tokenizer can be represented by an ordered list of merge rules m (1) , . . . , m ( M ) . For each time step t ∈ [ M ] , we apply all preceding merge rules m (1) , . . . , m ( t -1) to our corpora D i and use c ( t ) i,p to denote how many times the token pair p occurred in the partially merged text. We know that when the tokenizer was trained, the pair m ( t ) was more frequent at time t than any other pair. In other words,

̸

Collecting these constraints for all t and p defines a set of possible α 's.

Of course, because we only have samples from the category distributions and not the exact data the tokenizer was trained on, the above linear program may not be feasible, as the counts will include some sampling noise. To address this, we relax the constraints by introducing new non-negative variables v ( t ) for all t ∈ [ M ] , and v p for all pairs p , which represent the degree of constraint violation for each merge and pair, respectively. We replace our constraints with new ones of the form

̸

In general, we expect v ( t ) to be large when m ( t ) is over-represented in the tokenizer training data and v p to be large when p is over-represented in the mixture defined by α . This new system of constraints is guaranteed to be feasible, as the v 's can be made arbitrarily large. To produce the best possible estimate, our objective is to minimize the total constraint violation ∑ M t =1 v ( t ) + ∑ p v p . We call the resulting linear program LP1 . To estimate α , we solve LP1 and report the optimal value of α as ˆ α .

̸

As written, LP1 can be prohibitively large. If our vocabulary has size V , the total number of constraints scales like O ( V 3 ) since there are O ( V ) time steps t to consider and O ( V 2 ) competing byte pairs p = m ( t ) . Additionally, there are O ( V 2 ) variables v p . The first step to reduce the size is to limit t to the first T merges. We will call this truncated program LP1 T . However, even for modest choices of T , LP1 T can still have millions of variables and tens of billions of constraints. In the

following sections, we will describe how to efficiently solve LP1 T using simultaneous delayed row (constraint) and column (variable) generation [12, 23].

3.2 Efficient storage of pair counts

First, as a preprocessing step, we apply the target tokenizer to each language corpus D i , recording the pair counts c ( t ) i,p after each merge is applied for later use. Naively, this would require a large amount of space, since the number of possible pairs p scales like O ( V 2 ) . However, note that c ( t ) i,p = c ( t +1) i,p only when p overlaps with m ( t ) . In other words, pairs with no overlap with the most recent merge will have unchanged counts. Thus, there are only O ( V ) differences between c ( t ) i, · and c ( t +1) i, · . In practice, the number of changes caused by a single merge is usually a few hundred at most. By saving only the incremental changes from each set of pair counts to the next, we can efficiently record the pair counts at every iteration of the tokenization process.

3.3 Efficient constraint violation detection

Our plan is to solve LP1 T using only a subset of its constraints, giving a potential solution ( α, v ) . We then check whether ( α, v ) violates any of the constraints of LP1 T and add any such constraints to the subset. This requires an efficient method to detect violated constraints, which we describe below.

̸

For convenience, let s ( t ) p := ∑ n i =1 α i c ( t ) i,p and recall that, for a given time step t , we want to check whether v ( t ) + s ( t ) m ( t ) ≥ max p ( s ( t ) p -v p ) . Naively, we would do so by iterating over all possible p = m ( t ) to see if the constraint is violated, which can be quite costly. Moreover, we must do this for all t ≤ T . However, by taking advantage of the structure of the s ( t ) p as t varies, we can reduce our work substantially.

The first step is to take each initial pair p and add it to a priority queue with priority s (0) p -v p . This can be done in aggregate in O ( V 2 ) time using a fast heap building algorithm. Now, we can iterate through the pairs in descending s (0) p -v p order using the queue's delete-min operation. For each pair p , we can check whether v (0) + s (0) m (0) > s (0) p -v p and if not, we mark the corresponding constraint as violated. Once we find a p sat satisfying the constraint, we stop, since all pairs remaining in the queue must satisfy their constraints. If there were k pairs before p sat in the queue, then the total time taken is O ( k log V ) .

Crucially, we can quickly update the priority queue to reflect the state at t = 1 . Since we precomputed all count changes from c (0) i,p to c (1) i,p , we know what queue entries need to be updated or inserted. If k new new pairs were created and k old pairs had their counts changed when pair m (0) was merged, then we can update the priority queue using k new insert operations and k old decrease-priority operations, which can be done in O (( k new + k old ) log V ) time. Now that we have updated the priority queue, we can repeat the above procedure to check for any constraint violations for t = 1 . By iterating this process, we can quickly check for violated constraints for all t ≤ T .

3.4 Lazy variable and constraint generation

Now we are ready to efficiently solve LP1 T . We begin by guessing uniform proportions for α , and v ( t ) = v p = 0 for all t, p . Then we use our constraint checker to identify violated constraints of LP1 T and construct a lazy version of LP1 T , which we denote LP2 T , using only those constraints and the variables they contain. We then solve LP2 T , which gives us a new guess for α . We repeat the above steps, adding progressively more constraints (along with their variables) to LP2 T until we find a solution that is also feasible for LP1 T . It follows that this solution is optimal for LP1 T since the two programs share the same objective. This is guaranteed to happen eventually because there are a finite number of constraints and variables to add.

In practice, the constraint violation detection can typically check T = 30000 merges in less than 10 seconds. On difficult instances, such as those for commercial tokenizers in §5, the full solve can take up to a day to complete. Easier instances like those in Table 1 can be solved in a few minutes.

̸

4 Experiments

In our initial experiments, we train tokenizers on known data mixtures and measure the accuracy of our attack's prediction. We consider mixtures of natural languages, programming languages, and data sources (which we also refer to as domains).

4.1 Setup

Because BPE tokenizers operate on bytes, we measure the proportion of each category in terms of bytes. Each tokenizer is trained on a mixture of n categories, where n varies from 5 and 112. We randomly sample the n categories and their weights from the unit simplex (using the algorithm from [53]), and train 100 tokenizers on 10 GB of data. The data for each category is sampled from the corresponding corpus; if there is not enough data for any category (e.g., we have many low-resource languages), we duplicate the data until the necessary amount is achieved, to preserve the desired mixture ratio. We train tokenizers using the HuggingFace tokenizers library with a maximum vocabulary size of 30,000, and apply a minimal set of common pretokenization operations: we split on whitespace and only allow digits to be merged with other contiguous digits.

After training the tokenizers, we apply our attack. We estimate merge frequencies for each category by sampling 1 GB of data per category, or less if there is not that much data. Note that the data used for training the tokenizer and estimating pair frequencies are sampled from the same distribution, but are not necessarily the same data. We use mean squared error to evaluate the estimated proportions, MSE := 1 n ∑ n i =1 (ˆ α i -α ∗ i ) 2 . In practice, we report log 10 (MSE) .

In §B.4, we analyze how our attack's performance varies with the amount of data used and the number of merges T considered from the merge list.

Natural Language Mixtures We use the Oscar v23.01 corpus [1], which is based on the Nov/Dec 2022 dump from Common Crawl. We consider the 112 languages with at least 1 MB of data.

Programming Language Mixtures We use the GitHub split of RedPajama [22]. To determine the programming language for each record, we map the file extension to its associated language (e.g., .py → Python ). This leads to a total of 37 programming languages.

Domain Mixtures Weconsider the following five English domains (adapted from [38]), instantiated by data from the RedPajama dataset: Wikipedia , containing English Wikipedia dumps from Jun-Aug 2022, Web , Common Crawl data that was de-duplicated and filtered for English, Books from the Gutenberg Project and Books3 of The Pile, Code from GitHub, and Academic , which contains LaTeX files of scientific papers on ArXiv.

For dataset details, e.g., the full list of categories and data sizes, please see §B.

4.2 Baselines

We construct two intuitive baselines: one based on tokenizer encoding efficiency and one based on analysis of tokens in the vocabulary. Neither takes the BPE training algorithm into account.

Baseline based on tokenizer encoding efficiency (TEE) Intuitively, we expect data categories with greater representation in the training data to be encoded more efficiently by the resulting tokenizer. To capture this, we calculate a given tokenizer's byte-to-token ratio on each category (a more efficient tokenizer will encode more bytes per token), then normalize it by that of a reference tokenizer trained on only that category, to control for different categories being inherently easier or harder to encode. Then we learn a log-log linear model to predict each category's true proportion given the encoding efficiency. To ensure correctness, we normalize the resulting set of predictions into a probability distribution.

Baseline based on token classification (TC) We also consider a baseline that assigns each token in the vocabulary to a data category based on its empirical frequency in the sample data. Intuitively, we expect that if there is a large proportion of e.g., English data in the training data, then there will be more 'English' tokens. For each token in the vocabulary, we count its occurrences in data sampled from each category, and assign it to the one in which it is most frequent (we find that hard assignment outperforms all variations of soft assignment). Then we count the number of tokens assigned to each category, and normalize the counts to produce an estimate of the data mixture.

4.3 Results

Shown in Table 1, our attack is highly effective. Over all mixture types and values of n , we achieve mean MSE two and six orders of magnitude better than random guessing. In contrast, the baselines based on tokenizer efficiency (TEE) and token classification (VC) do not come close to the kind of precision possible with our attack, achieving at best one order of magnitude better than random.

We observe that the setting with the highest attack success is mixed languages, whereas the most challenging is mixed English domains. This is perhaps unsurprising when considering the source of signal for our attack, which is the different token pair frequencies in different data categories. Intuitively, we would expect these to be very different for different natural languages, which have distinct vocabularies. In contrast, programming languages can share many syntactic features, such as using indents, curly brackets {} , and English variable names. Even more so, English data from different domains (e.g., books vs. Wikipedia) will largely share the same vocabulary but have subtle differences in token frequencies due to style, topic, and formatting. Nonetheless, even in this most challenging setting, we achieve accuracy 100 × better than either baseline.

5 Attacking commercial tokenizers

After validating our attack in synthetic experiments (§4), we apply it to infer training data mixtures of off-the-shelf commercial tokenizers. We refer to tokenizers by the name of the model they were first released with, whose pretraining data they most likely reflect. We consider GPT-2 [47], GPT-3.5 [45], GPT-4O [46], LLAMA [56], LLAMA 3 [37], MISTRAL [4], MISTRAL-NEMO [6], GPT-NEOX [13], CLAUDE [8], and GEMMA [55]. While many of these are closed models, their tokenizers are publicly available so that customers can estimate the cost of queries ahead of time. We note that the

LLAMA, GEMMA, and MISTRAL tokenizers use characters instead of bytes as the base vocabulary for the BPE algorithm; this does not affect our attack, but we discuss the distinction in §C.6.

In these experiments, we aim to infer the proportion of different natural languages, code, and English domains, for a total of 116 categories. We consider code as a single category (not split into separate programming languages) because some languages like Markdown and Pod6 are almost entirely English, and we do not expect the distribution of programming languages in pretraining to differ substantially from that of GitHub, the largest public code hosting platform. To infer the distribution of English domains, we replace the English category with the four English domains from §4 (web, books, Wikipedia, and academic), which we expect to approximately cover the English data.

Our predictions are shown in Figure 2, with specific numbers in §C.1. Below, we discuss our findings in comparison with publicly disclosed information about these models.

5.1 GPT models

All tokenizers accompanying GPT models are open-source on tiktoken . 4 There are three such tokenizers, released with GPT-2, GPT-3.5, and the very recent GPT-4O. 5

GPT-2 GPT-2 was trained on WebText, consisting of text scraped from outbound links from Reddit, and filtered to be English-only. The GPT-2 tokenizer was reused for GPT-3. Indeed, we confirm the training data consists of 99.1% English. However, we surprisingly estimate that only 83.6% of the data was web, with another 15.4% being books, which were not explicitly included in WebText. In a data contamination analysis, the authors indeed report that they find books in WebText, but our estimate suggests the contamination may be deeper. We note that books were a popular source of pretraining data for early Transformer LMs, with GPT-1 being trained entirely on BooksCorpus [63].

GPT-3.5 The GPT-3.5 family of models is known to depart from its predecessors by training on large amounts of code: the first model in this family was code-davinci-002 , trained on text and code. In fact, some evidence suggests that GPT-3.5's large leap in reasoning abilities comes from this code data, which intuitively requires similar procedural skills [26]. The GPT-3.5 tokenizer was reused for GPT-4.

Indeed, we estimate that GPT-3.5 is trained on 62.6% code. In the domain breakdown, 27.3% is of the data is web, 6.8% books, and 0.2% academic articles. The substantial representation of books (though lower than GPT-2) is consistent with findings that this model has memorized a wide collection of copyrighted books [20].

GPT-4O GPT-4O is a multimodal model announced as more multilingual than its predecessors; its tokenizer achieves a better compression rate on non-English languages, and the model has notably better non-English performance.

Our findings support this. GPT-4O is trained on 39.0% non-English text, compared to only 3.2% for GPT-3.5. The language distribution has a thick non-English tail, with 68 languages that make up at least 0.1% of the data: the most common are French (2.9%), Russian (2.8%), Spanish (2.8%), Portuguese (2.3%), Dutch (2.0%), German (1.8%), Arabic (1.6%), and Hindi (1.4%). Additionally, GPT-4O was trained on 7.4% books.

5.2 LLAMA MODELS

LLAMA The training data for LLAMA is known to be primarily English, though the Wikipedia split ' covers 20 languages which use either the Latin or Cyrillic scripts: bg , ca , cs , da , de , en , es , fr , hr , hu , it , nl , pl , pt , ro , ru , sl , sr , sv , uk '. The training data is reportedly sourced from Common Crawl (67% of examples), C4 (15.0%), Github (4.5%), Wikipedia (4.5%), Books (4.5%), ArXiv (2.5%), and StackExchange (2.0%). The LLAMA tokenizer was reused for LLAMA2.

We corroborate that LLAMA is indeed primarily made up of the stated languages; when combined with code, this sums to 95.7% of the training corpus. Indeed, other generally high-resource languages, such as Chinese, Arabic, and Hindi, have 0.0% representation. However, we predict a very different domain distribution compared to what is reported in the paper for LLAMA's pretraining data. We predict that the tokenizer is trained on 23.1% books, 11.3% web, 6.7% Wikipedia, and 8% ArXiv. The high representation of books is surprising - we hypothesize that the tokenizer was trained on a different distribution than the LM, primarily focusing on books which uses a more standard vocabulary compared to web data.

LLAMA 3 We observe that LLAMA 3, rather than training a new tokenizer, extends GPT-3.5's merge list (of 100,000 merges) with an extra 27,744 merges. Thus, we apply our attack to these new merges to infer what data was used to extending the LLAMA 3 tokenizer. It is reported that LLAMA 3 is trained on more than 5% of ' high-quality non-English text that covers over 30 languages .' We find that LLAMA 3 is extended with primarily non-English text (48.5%) and code (30.2%), indicating that the goal of extending GPT-3.5's tokenizer was primarily for multilingual use.

5.3 MISTRAL

MISTRAL MISTRAL models ' handle English, French, Italian, German and Spanish ' [5]; indeed, we find that these are the top five languages in the training data. There is a long tail of other languages, but they predominantly (97%) use either the Latin or Cyrillic script.

MISTRAL NEMO In contrast, MISTRAL NEMO was ' designed for global, multilingual applications, bringing frontier AI models to... all languages .' It introduces a new tokenizer (based on tiktoken instead of sentencepiece ), which is the most multilingual of tokenizers we study, training on 46.6% non-English text. French (6.3%) and Arabic (4.8%) are the most common non-English languages.

5.4 GPT-NEOX

The tokenizer of GPT-NEOX [14] was trained on the Pile [29] with ' certain components... upsampled .' It is popularly re-used in open-source model development, including by OLMO [31], PYTHIA [13], and DCLM [34]. Though our domains do not map neatly onto the constituent datasets of the Pile, our inference is generally consistent, with a prediction of 43.7% web, 26.3% books, 12.1% academic, 15.2% code, and 2.7% non-English text.

5.5 GEMMA

GEMMA [55] is reported as training on ' primarily-English data from web documents, mathematics, and code .' Gemma uses a subset of the Gemini tokenizer; we believe it is likely that this subset was obtained by truncation, which would make our inferences valid for Gemini as well. We predict that GEMMA is trained on 45.7% English, which comes from 25.6% web, 12.8% books, 4.3% academic, and 3.0% Wikipedia. It is also trained on 28.4% non-English text, which explains its large multilingual vocabulary of 256,000 tokens. However, compared to GPT-4O, the multilingual representation is more skewed toward languages that use Latin or Cyrillic scripts.

5.6 CLAUDE

Very little is known about models from the CLAUDE family, but a remark in the Anthropic SDK suggests that CLAUDE 1 [8] and 2 [9] share the same tokenizer, which is open-source, while CLAUDE 3 [10] uses a different (closed) tokenizer. Our attack predicts that CLAUDE was trained on 57.5% code, 38.8% English, and 3.7% other languages. Moreover, half of its data (17.4% overall) comes from books, with substantial contribution from Wikipedia (3.7%) and academic text (5.1%) as well. The lack of multilingual training data likely explains why a new tokenizer was trained for CLAUDE 3, which boasts ' increased capabilities... in non-English languages ' [10].

6 Robustness analysis

6.1 Is the attack robust to distribution shift?

We measure the impact of using out-of-distribution data instead of data from the tokenizer's training distribution to count merge frequencies. Note that in the main experiments, we show that the attack is effective at separating English domains, so performance degradation under distribution shift is expected. To empirically measure this, we train tokenizers on mixtures of n = 10 languages using web data from Oscar, but estimate merge frequencies using corresponding language splits of Wikipedia, a substantially different domain. Using the same settings as the main experiments (§4), we achieve log MSE of -3 . 53 , compared to -7 . 66 with no shift. Thus, the attack performance drops considerably under this extreme distribution shift, while remaining 100 × better than random.

6.2 Is the attack robust to unaccounted-for categories?

In a typical attack setting, the attacker may not have explicitly accounted for every source of data used to train the tokenizer. To show that our approach is robust in the presence of such data, we modify our n = 112 languages experiment from our main experiments (§4). We randomly withhold a random subset of 1 to 50 languages from the solver and measure the resulting prediction error on the remaining languages. Results are shown in Figure 4. Although the performance of our method does worsen as the amount of unknown data increases, the predictions remain substantially better than random.

7 Related work

Attacks on LMs Membership inference, the task of inferring whether a particular example was part of the training data, has been studied in great depth for language models [15, 17, 40, 42, 24, 51]. Attacks commonly use model-assigned probabilities, potentially with another reference model, to make an inference. The problem remains extremely difficult, with a recent survey finding that many attacks perform near random when pretraining data is deduplicated, as in common practice [24].

Closely related to this, some memorization attacks aim to extract memorized examples from the pretraining data via prompting attacks [18, 44]. Recently, there has even been progress in recovering the parameters of black-box LMs through model stealing attacks [25, 19].

Distribution inference In contrast to membership inference, distribution inference is concerned with inferring global properties of a model's training data, but has not previously been studied for LM pretraining data. In early works, these attacks were successful against machine learning classifiers [11, 28], CNNs [54], and GANs [62]. More recently, some works show that multi-party machine learning can leak property information such as which authors contributed to the data [41, 61]. All previous distribution inference attacks take a meta-classifier approach, where models are trained on datasets with different properties, then a meta-classifier is trained using those models.

8 Conclusion

In this work, we present a data mixture inference attack that solves for the distributional make-up of a tokenizer's training data, which is commonly representative of the language model's pretraining data. Beyond the properties we study, we believe there is still a wealth of information hidden in tokenizer merge lists. This can shed light on the secretive and often contentious design decisions surrounding pretraining data today, potentially enabling external auditing for safety, copyright issues, and distributional biases. We hope our work will inspire continued research into inferring more global properties of training data, for tokenizers and language models more generally.

3

10

4

10

3

10

2

10

1

10

8

10

7

10

6

10

5

10

4

10

MSE

Random Guess

Acknowledgments

We would like to thank Luca Soldaini for identifying the cause of redundant merges in some commercial LLM tokenizers (discussed in §C.3), and Jiacheng Liu, Orevaoghene Ahia, Xiaochuang Han, Muru Zhang, Thao Nguyen, Scott Geng, Rulin Shao, Zhaofeng Wu, and the greater UW NLP community for valuable feedback and conversations on this work. We are also grateful to the HuggingFace user Xenova for posting tokenizers -compatible versions of many tokenizers. This work is supported by Microsoft Grant for Customer Experience Innovation, the National Science Foundation under grant No. 2019844, 2112471, and 2229876 and DMS-2134012. Both co-first authors (JH and AL) are supported by the NSF Graduate Research Fellowship Program.

References

J. Abadji, P. Ortiz Suarez, L. Romary, and B. Sagot. Towards a cleaner document-oriented multilingual crawled corpus. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis, editors, Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages 4344-4355, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022.lrec-1.463 .

O. Ahia, S. Kumar, H. Gonen, J. Kasai, D. Mortensen, N. Smith, and Y. Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9904-9923, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.614. URL https:// aclanthology.org/2023.emnlp-main.614 .

O. Ahia, S. Kumar, H. Gonen, V. Hoffman, T. Limisiewicz, Y. Tsvetkov, and N. A. Smith. Magnet: Improving the multilingual fairness of language models with adaptive gradient-based tokenization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://arxiv.org/abs/2407.08818 .

M. AI. Mistral 7b, 2023. URL https://mistral.ai/news/announcing-mistral-7b/ .

M. AI. Au large, 2024. URL https://mistral.ai/news/mistral-large/ .

M. AI. Mistral nemo, 2024. URL https://mistral.ai/news/mistral-nemo/ .

A. Albalak, Y. Elazar, S. M. Xie, S. Longpre, N. Lambert, X. Wang, N. Muennighoff, B. Hou, L. Pan, H. Jeong, C. Raffel, S. Chang, T. Hashimoto, and W. Y. Wang. A survey on data selection for language models, 2024. URL https://arxiv.org/abs/2402.16827 .

Anthropic. Introducing claude, 2023. URL https://www.anthropic.com/news/ introducing-claude .

Anthropic. Claude 2, 2023. URL https://www.anthropic.com/news/claude-2 .

Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic. com/news/claude-3-family .

G. Ateniese, L. V. Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. Int. J. Secur. Netw. , 10(3):137-150, sep 2015. ISSN 1747-8405. doi: 10.1504/IJSN.2015.071829. URL https://doi.org/10.1504/IJSN.2015.071829 .

J. Benders. Partitioning procedures for solving mixed-variables programming problems. Numerische mathematik , 4(1):238-252, 1962.

S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal. Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning , ICML'23. JMLR.org, 2023.

S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022. URL https://arxiv.org/abs/2204.06745 .

N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2633-2650. USENIX Association, Aug. 2021. ISBN 978-1-939133-24-3. URL https://www.usenix. org/conference/usenixsecurity21/presentation/carlini-extracting .

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1897-1914. IEEE, 2022.

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramèr. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1897-1914, 2022. doi: 10.1109/SP46214.2022.9833649.

N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=TatRHT_1cK .

N. Carlini, D. Paleka, K. D. Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy, E. Wallace, D. Rolnick, and F. Tramèr. Stealing part of a production language model, 2024.

K. Chang, M. Cramer, S. Soni, and D. Bamman. Speak, memory: An archaeology of books known to ChatGPT/GPT-4. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7312-7327, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.453. URL https://aclanthology.org/2023.emnlp-main.453 .

C. A. Choquette-Choo, F. Tramer, N. Carlini, and N. Papernot. Label-only membership inference attacks. In International conference on machine learning , pages 1964-1974. PMLR, 2021.

T. Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL https://github.com/togethercomputer/RedPajama-Data .

G. B. Dantzig and P. Wolfe. Decomposition principle for linear programs. Operations Research , 8(1):101-111, 1960. URL https://www.jstor.org/stable/167547 .

M. Duan, A. Suri, N. Mireshghallah, S. Min, W. Shi, L. Zettlemoyer, Y. Tsvetkov, Y. Choi, D. Evans, and H. Hajishirzi. Do membership inference attacks work on large language models?, 2024. URL https://arxiv.org/abs/2402.07841 .

M. Finlayson, X. Ren, and S. Swayamdipta. Logits of api-protected llms leak proprietary information, 2024.

H. Fu, Yao; Peng and T. Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's Notion , Dec 2022. URL https://yaofu.notion.site/ How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sou

P. Gage. A new algorithm for data compression. The C Users Journal archive , 12:23-38, 1994. URL https://api.semanticscholar.org/CorpusID:59804030 .

K. Ganju, Q. Wang, W. Yang, C. A. Gunter, and N. Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security , CCS '18, page 619-633, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356930. doi: 10.1145/3243734.3243834. URL https://doi.org/10. 1145/3243734.3243834 .

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027 .

J. Geiping, A. Stein, M. Shu, K. Saifullah, Y. Wen, and T. Goldstein. Coercing llms to do and reveal (almost) anything, 2024. URL https://arxiv.org/abs/2402.14020 .

D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. Smith, and H. Hajishirzi. OLMo: Accelerating the science of language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15789-15809, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.841 .

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www. gurobi.com .

S. Land and M. Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models, 2024. URL https://arxiv.org/abs/2405.05417 .

J. Li, A. Fang, G. Smyrnis, M. Ivgi, M. Jordan, S. Gadre, H. Bansal, E. Guha, S. Keh, K. Arora, S. Garg, R. Xin, N. Muennighoff, R. Heckel, J. Mercat, M. Chen, S. Gururangan, M. Wortsman, A. Albalak, Y. Bitton, M. Nezhurina, A. Abbas, C.-Y. Hsieh, D. Ghosh, J. Gardner, M. Kilian, H. Zhang, R. Shao, S. Pratt, S. Sanyal, G. Ilharco, G. Daras, K. Marathe, A. Gokaslan, J. Zhang, K. Chandu, T. Nguyen, I. Vasiljevic, S. Kakade, S. Song, S. Sanghavi, F. Faghri, S. Oh, L. Zettlemoyer, K. Lo, A. El-Nouby, H. Pouransari, A. Toshev, S. Wang, D. Groeneveld, L. Soldaini, P. W. Koh, J. Jitsev, T. Kollar, A. G. Dimakis, Y . Carmon, A. Dave, L. Schmidt, and V. Shankar. Datacomp-lm: In search of the next generation of training sets for language models, 2024. URL https://arxiv.org/abs/2406.11794 .

M. Li, S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022. URL https://arxiv.org/abs/2208.03306 .

T. Limisiewicz, T. Blevins, H. Gonen, O. Ahia, and L. Zettlemoyer. MYTE: Morphology-driven byte encoding for better and fairer multilingual language modeling. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 15059-15076, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. URL https://aclanthology.org/ 2024.acl-long.804 .

A. . M. Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/ 2407.21783 .

S. Longpre, G. Yauney, E. Reif, K. Lee, A. Roberts, B. Zoph, D. Zhou, J. Wei, K. Robinson, D. Mimno, and D. Ippolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. 2023. URL https://arxiv.org/abs/2305. 13169 .

Y. MA, Y. Liu, Y. Yu, Y. Zhang, Y. Jiang, C. Wang, and S. Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=KIPJKST4gw .

J. Mattern, F. Mireshghallah, Z. Jin, B. Schoelkopf, M. Sachan, and T. Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages 11330-11343, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.719. URL https://aclanthology.org/2023.findings-acl.719 .

L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 691706, 2019. doi: 10.1109/SP.2019.00029. URL https://ieeexplore.ieee.org/document/ 8835269 .

F. Mireshghallah, K. Goyal, A. Uniyal, T. Berg-Kirkpatrick, and R. Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 8332-8347, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.570. URL https://aclanthology.org/2022.emnlp-main.570 .

N. Mireshghallah, N. Vogler, J. He, O. Florez, A. El-Kishky, and T. Berg-Kirkpatrick. Simple temporal adaptation to changing label sets: Hashtag prediction via dense KNN. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 7302-7311, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.452. URL https://aclanthology.org/2023.emnlp-main.452 .

M. Nasr, N. Carlini, J. Hayase, M. Jagielski, A. F. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable extraction of training data from (production) language models, 2023. URL https://arxiv.org/abs/2311.17035 .

OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/index/chatgpt/ .

OpenAI. Hello GPT-4o, 2024. URL https://openai.com/index/hello-gpt-4o/ .

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019. URL https://cdn.openai.com/ better-language-models/language_models_are_unsupervised_multitask_ learners.pdf .

J. Rumbelow and M. Watkins. Solidgoldmagikarp (plus, prompt generation), 2023. URL https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/ solidgoldmagikarp-plus-prompt-generation .

A. Schäfer, S. Ravfogel, T. Hofmann, T. Pimentel, and I. Schlag. Language imbalance can boost cross-lingual generalisation, 2024. URL https://arxiv.org/abs/2404.07982 .

R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In K. Erk and N. A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715-1725, Berlin, Germany, Aug. 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162 .

W. Shi, A. Ajith, M. Xia, Y. Huang, D. Liu, T. Blevins, D. Chen, and L. Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=zWqr3MQuNs .

R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

N. A. Smith and R. Tromble. Sampling uniformly from the unit simplex. Technical report, 2004. URL https://www.cs.cmu.edu/~nasmith/papers/smith+tromble.tr04.pdf .

A. Suri and D. Evans. Formalizing and estimating distribution inference risks. In Privacy Enhancing Technologies Symposium , volume 2022, pages 528-551, 2022. URL https:// petsymposium.org/popets/2022/popets-2022-0121.php .

G. Team. Gemma: Open models based on gemini research and technology, 2024.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023. URL https://arxiv.org/abs/2302.13971 .

J. Wang, T. Gangavarapu, J. N. Yan, and A. M. Rush. Mambabyte: Token-free selective state space model, 2024. URL https://arxiv.org/abs/2401.13660 .

B. Workshop. Bloom: A 176b-parameter open-access multilingual language model, 2023. URL https://arxiv.org/abs/2211.05100 .

Y. Xie, A. Naik, D. Fried, and C. Rose. Data augmentation for code translation with comparable corpora and multiple references. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 13725-13739, Singapore, Dec. 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.917. URL https://aclanthology.org/2023.findings-emnlp.917 .

J. Yang. Rethinking tokenization: Crafting better tokenizers for large language models, 2024. URL https://arxiv.org/abs/2403.00417 .

W. Zhang, S. Tople, and O. Ohrimenko. Leakage of dataset properties in Multi-Party machine learning. In 30th USENIX Security Symposium (USENIX Security 21) , pages 2687-2704. USENIX Association, Aug. 2021. ISBN 978-1-939133-24-3. URL https://www.usenix. org/conference/usenixsecurity21/presentation/zhang-wanrong .

J. Zhou, Y. Chen, C. Shen, and Y. Zhang. Property inference attacks against gans. In 29th Annual Network and Distributed System Security Symposium, NDSS 2022, San Diego, California, USA, April 24-28, 2022 . The Internet Society, 2022. URL https://www.ndss-symposium.org/ ndss-paper/auto-draft-240/ .

Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV) , December 2015.

A Discussion of possible defenses

We discuss some possible approaches for defenses to our attack.

Post-hoc changing the order of merge rules Model producers may consider changing the order of merge rules, which is the source of signal for our attack, after the tokenizer is trained. However, naively re-ordering merge rules for a tokenizer would be damaging, as it can lead to unfamiliar encodings of words as well as entirely unreachable tokens. The only functionally-equivalent reordering would be within contiguous sections of merge rules, where each token appears exclusively on the left or right side of all merges it appears in. In this case, we can easily adapt our method by working at the level of contiguous non-conflicting sections instead of individual merge rules, as we know that each section has higher frequencies than the next.

Hiding pretokenization rules Our method relies on a reasonable reconstruction of the pretokenization rules, which control what kinds of merge rules are considered. It is not necessary to share pretokenization rules, as they are not strictly necessary for inference. However, we find that important pretokenization rules (like whether to pre-tokenize on spaces, digits, and punctuation) are easy to infer from manual inspection.

Not using BPE tokenizers Model producers may choose to forgo BPE tokenizers entirely in future models. Despite the current popularity of BPE, there is a lively area of research into alternative methods of tokenization [57, 60, 36, 3]. While we only explore BPE tokenizers in this paper, it is plausible that any tokenizer learning algorithm will leak information about its training data, as they are specifically developed to best encode the given data.

B Experiment details & additional results

B.1 Data details

The full set of categories that we use in §4 and the amount of data available for each category are shown in Table 5, Table 6, and Table 7.

Language mixtures We use OSCAR-23.01, which is the January 2023 version of the OSCAR Corpus based on the November/December 2022 dump of Common Crawl. We only keep languages with at least 1 MB of data.

Code mixtures We use the GitHub split of RedPajama-Data-1T, which is an open reproduction of LLAMA's training data.

Domain mixtures We use five splits of RedPajama, namely Wikipedia, Common Crawl, Books, Github, and ArXiv. To reduce disk usage, we download only 8% of the CC URLs.

Below, we enumerate the licenses for these datasets.

Oscar: CC0 1.0 Universal

RedPajama has different licenses for each subset

-C4: ODC-BY

-GitHub: MIT, BSD, or Apache

-Books3: MIT

-Project Gutenberg: Apache 2.0

-ArXiv: CCo 1.0

-Wikipedia: CC-BY-SA-3.0

B.2 Compute details

We run all of our experiments on CPUs. For training tokenizers and calculating pair frequencies, we use 16-32 CPUs and a variable amount of memory (ranging from 4 GB to 64 GB) depending on the data. Training a tokenizer on 10 GB of data (as in our experiments) usually takes around 10 minutes, while calculating pair counts takes between 1 minute and 2 hours, again depending on the data. To solve our linear programs, we use Gurobi [32].

B.3 Baseline further discussion

For the baseline based on tokenizer efficiency, we plot the relationship between the true training proportion and tokenizer efficiency (as the normalized byte-to-token ratio) in Figure 5. As expected, the more data for a particular language there is in training, the more efficiently the tokenizer encodes that language. However, the correlation is clearly imprecise, with the true proportion ( x -axis) varying up to an order of magnitude given the encoding efficiency ( y -axis).

Between the baselines based on tokenizer encoding efficiency (TEE) versus vocabulary item categorization (VIC), we find that TEE performs better for mixtures of code, while VIC performs better for mixtures of languages. This makes sense, because different languages have very distinct vocabularies, while vast inherent differences between languages may make encoding efficiency hard to compare, even when using normalization.

B.4 Scaling Analysis

Using our setup for controlled experiments (§4), we analyze how our attack's performance varies with the amount of data used (§B.4.1) and the number of merges we consider from the merge list (§B.4.2).

B.4.1 How many data samples should we use from each category?

We explore how the attack's performance scales with the amount of data sampled from each distribution D i for calculating pair counts. For each type of mixture considered in §4, we train 100

10

3

10

2

10

1

Proportion in training

10 0

4 × 10

1

6 × 10

1

Tokenizer encoding efficiency

10 6

10 7

10 8

10 9

10 10

Bytes of Data

10

6

10

4

10

2

MSE

Mixture Type

languages

code

domains

10 2

10 3

10 4

Number of Merges

T

10

8

10

7

10

6

10

5

10

MSE

n

5

10

30

112

4

new tokenizers considering only categories with at least 10 GB of data available. For our attack, we compare sampling 1 MB, 10 MB, 100 MB, 1 GB, and 10 GB of data for each category, and use T = 3000 merges. Shown in Figure 6, more data consistently improves the accuracy of predictions.

B.4.2 How many merges should we consider?

Next, we investigate how performance scales with the number of merges T that we apply from the merge list. Using the same 100 tokenizers from §4, we solve for the data mixture using various choices of T ∈ [30 , 30000] . Shown in Figure 7, we find that when there are more categories, it is useful to consider more merges. This makes sense because more constraints may be needed to bound the solutions in higher dimensions.

B.4.3 Scaling analysis under distribution shift

Here, we repeat the distribution shift experiments in §6.1 while varying the amount of data and merges used. Shown in Figure 8 and Figure 9, we find a U-shaped curve for how performance scales with the amount of data used for calculating pair frequencies, unlike our main experiments in §4.

10 6

10 7

10 8

10 9

10 10

Bytes of data

10

3

MSE

# merges T

100

300

1000

3000

10000

29772

C Commercial tokenizers

In this section, we provide more detailed results and discussion of commercial tokenizers.

C.1 Full results for commercial tokenizers

We report the full inferences for commercial tokenizers (§5) over 116 categories (111 languages, 4 English domains, and code) in Table 4.

C.2 Snapshot of commercial tokenizer merge lists

We show the first 50 merges of the commercial tokenizers we study in Table 2 for qualitative inspection.

C.3 Handling redundant merges

We observe that the merge list of LLAMA, LLAMA 3, GEMMA, and MISTRAL contain clusters of redundant merge rules. For instance, in the LLAMA 3 merge list, we see the sequence of merges _ the , _t he , and _th e , as well as _ and , _a nd , and _an d . Because the merge path for every token is unique, it is impossible for more than one of these merges to ever be used, and we empirically verify this by applying the tokenizer to a large amount of text.

We find that this is an artifact of the conversion from sentencepiece to Huggingface tokenizers format. To construct the merge list, the conversion algorithm naively combines every pair of tokens in the vocabulary, and then sorts them by token ID, which represents order of creation. While this is functionally correct, because the redundant merges are not products of the BPE algorithm (i.e., they do not actually represent the most-likely next-merge), we need to remove them for our algorithm. To do this, we do some simple pre-processing: for every cluster of redundant merges, we record the path of merges that achieves each merge; the earliest path is the one that would be taken, so we keep that merge and remove the rest.

As an aside, this means that a tokenizer's merge list can be completely reconstructed from its vocabulary list ordered by token creation . Given only the resulting token at each time step, we can derive the corresponding merge.

C.4 Manual merges in GEMMA

We notice that the first 1,395 merges in GEMMA consistent entirely of merges of consecutive \ n , followed by merges of consecutive \ t , and finally merges of consecutive whitespace characters _ . They appear to be placed there manually and are not organically learned by the BPE algorithm, and do not correspond to increasing vocabulary IDs. Therefore, we remove these merges so that the remaining ordered merge rules align with monotonically increasing vocabulary ID.

We note that LLAMA and GEMMA, the two other tokenizers based on sentencepiece , also contain manually inserted merges of whitespace, but at the end of the merge list instead of the top. Since they are not within the top T = 30 , 000 merges we consider, we do not do any special pre-processing.

10 2

10 3

10 4

Number of merges

T

10

3

MSE

Bytes of data

10^6

10^7

10^8

10^9

10^10

C.5 GPT tokenizers

While GPT tokenizers are open source on tiktoken , they are not released in a format compatible with HuggingFace tokenizers. We used the tokenizers -compatible files uploaded by a HuggingFace user named Xenova . For instance, the GPT-4O tokenizer can be found at https://huggingface. co/Xenova/gpt-4o .

C.6 Discussion of sentencepiece tokenizers

The LLAMA and GEMMA tokenizers are trained with the sentencepiece library, which uses the same BPE algorithm, except the units of the base vocabulary are characters rather than bytes. In other words, the merge rules learned will apply to pairs of character sequences instead of byte sequences. While byte-level tokenizers always start with the same base vocabulary of 256 bytes, character-level tokenizers determine the base vocabulary using a character coverage hyperparameter (usually set to ∼ 0.9995), which determines what proportion of characters that appear in the training text will be in the base vocabulary. Byte fallback is used to represent the out-of-vocabulary characters using bytes.

To empirically test our attack on character-level BPE tokenizers, we train 100 sentencepiece tokenizers on mixtures of n = 10 natural languages. We apply our attack using the top T = 3000 merges, but otherwise use the same settings as §4.

With character-level tokenizers, we achieve an average log MSE of -4 . 09 compared to -1 . 39 for random guessing and -7 . 65 for byte-level tokenizers. That is, character-level tokenizers are harder for our algorithm to reverse than byte-level tokenizers! We believe this is because different languages have widely varying numbers of characters in their writing systems. For languages with many characters, their merges will appear lower in the merge list due to their lower average frequency compared to languages with fewer characters. This leads to a bias in representation among the first T merges considered by our approach.

C.7 Miscellaneous observation: tie-breaking in sentencepiece tokenizers

We observe that in sentencepiece tokenizers, after going deep in the merge list (about halfway), the merges begin forming groups, in which the length of the merge is ordered from shortest to longest. We trace this to how ties in pair counts are broken in sentencepiece , which is by length of the merge. In terms of reverse engineering, this points to a way to infer the size of the tokenizer training data, since exact ties in frequency become less likely as more training data is considered. We leave this direction to future work.

Table continues...

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims in the abstract and introduction are fully supported by evidence later in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: §A discusses potential defenses to our attack.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All information needed to reproduce our results are provided in the paper. We use open datasets and an open-source library to train our tokenizers.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We publicly release our code at https://github.com/alisawuffles/tokenizer-attack .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We state our choices for all parameters in the experiment sections (§4, §5) and provide analysis of how performance varies with these parameters in §6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard deviation of all of our results, as well as random baselines.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We discuss this in §B.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/ EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our knowledge, we conform to the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the introduction, we briefly discuss positive and negative implications of a data mixture inference attack.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite all creators or owners of the assets we use in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the task of  data mixture inference, i.e. trying to infer what kind of data (e.g. languages, code) a given LLM is trained on. They do so by leveraging the ordered merge rules learned by the LLM's BPE tokenizer. First, they formalize the problem. Next, authors propose a neatly explained solution using linear programming, with multiple computational optimizations which makes the inference feasible in practice. They then apply their method to a controlled setup, against models for which the data mixture is known and show that their method reaches a significantly better performance than random guessing. Lastly, they apply their method to widely used LLMs for which the mixture is not known and provide their inference results.\n\nSTRENGTHS:\n- Originality: The paper proposes the novel task of data mixture inference, and provides a compelling method to do so. \n- Quality: Both the task and proposed method is presented in a compelling way. The task of data mixture inference is formalized, and a novel, technically developed method is proposed. Both the evaluation in a controlled setup as well as the application of the method on other models for which the data mixture is unknown is convincing and interesting. From an experimental perspective, authors should provide a more sophisticated baseline than random guessing. Only then will the superiority of the proposed method become clear. Also, authors could elaborate more on potential defenses against the attack (see questions).\n- Clarity: the paper is written very clearly. \n- Significance: The paper proposes an attack able to discover a fundamental design decision made by model developers, which they likely want to keep proprietary. Exposing this attack, with its results is quite valuable to the field.\n\nWEAKNESSES:\n- Originality: NA\n- Quality: From an experimental perspective, authors should provide a more sophisticated baseline than random guessing. Only then will the superiority of the proposed method become clear. Also, authors could elaborate more on potential defenses against the attack (see questions).\n- Clarity: NA\n- Significance: A more elaborate discussion on potential defenses would increase the significance of the work's contribution.\n\nQUESTIONS:\n- When I think about the relevance of the findings represented in the paper, I believe that this mostly exposes a vulnerability to discover fundamental design decisions used by model developers, which they likely want to keep proprietary. From that perspective, could authors elaborate more on potential defenses and how they could perform? For instance, I do not fully understand why post-hoc re-ordering of the merge rules would hurt the utility of the tokenizer (maybe you can show this?). And if this is not a viable option, is the only defense then to not reveal any details on the tokenizer?\n- While the task at hand is novel, the only baseline considered (random guessing) is rather naïve. Could you include results of more elaborate baselines (e.g. leveraging the encoding efficiency of the tokenizer for each data sample? Or for the open source models, leveraging the perplexity of the model computed on the data samples)? This would shed more light into how difficult this problem might be and how valuable the proposed linear programming attack and its computational optimizations truly are. \n- One assumption that is made throughout the paper is that the attacker knows all potential data sources to include. Could you add an experiment on how the MSE changes when not all data sources are known (eg not including X natural languages, and have an increasing fraction of 'other data')?\n\nLIMITATIONS:\nAuthors should elaborate more on the potential defenses against this attack.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper proposes a novel data mixture inference attack to uncover the distributional makeup of pretraining data for large language models. By leveraging the ordered vocabulary learned by byte-pair encoding (BPE) tokenizers, the authors formulate a linear program to infer the relative proportions of different data categories (e.g., languages, programming languages, data sources) in the tokenizer's training set.

STRENGTHS:
1.	The paper introduces a previously understudied problem of data mixture inference, complementing existing membership inference attacks that focus on individual instances.
2.	The proposed attack method, based on BPE tokenizers' ordered merge rules, giving some insightful findings in inferring data mixture proportions.

WEAKNESSES:
1.	The attack relies on the specific behavior and structure of BPE tokenizers, which limits the generalizability of the method to different tokenizers.
2.	The attack hinges on the representativeness of tokenizer training data and the quality of sample data (from the same distribution in the paper), which may not always hold.

QUESTIONS:
1.	What the meaning of random guessing? Are there alternative methods to infer the proportions of different categories, possibly leveraging model performance metrics or behavioral patterns? 
2.	What strategies can be employed to ensure the quality of sample data for estimating pair frequencies.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors studied inference attack on large language models’ (LLM) tokenizer. In particular, they proposed an attack to inference training data sampling weight used to train tokenizer of LLMs, which usually is also the same sampling weight used to train the LLMs. They formulated their attack as a very large linear programming problem and proposed efficient methods to reduce the size of the linear programming and hence efficiently solve it. Moreover, they evaluated the proposed attack on tokenizers with known training data sampling weights and also some commercial LLMs.\n\nSTRENGTHS:\nThe proposed attack seems novel and the problem of interest is very relevant to key problems in LLM area. They verified the effectiveness of their attack on several data mix scenarios. The structure of the paper is also clear and easy to follow.\n\nWEAKNESSES:\nI listed a few directions that I think the paper can be improved.\n1. Theoretical analysis in this paper is a bit weak. The proposed algorithm in Section 3 does not provide convergence guarantee.  No upper bound provided for the size of the subset of constraints that are violated.  No analysis provided to support that the iterative process will finally provide an $\\alpha$ that violate no constraints in LP1.\n2. The attack implicitly assumes that one can access the corpus with similar words distribution as the pre-training datasets. However, it is unclear to me how different preprocessing techniques may affect the words distribution of the pre-training dataset and thus the effectiveness of the attack. It will be good to at least exam this impact via numerical experiments.\n\nQUESTIONS:\n1. Pretokenization and the starting vocabulary seems to be very influential on the effectiveness of the proposed algorithm. In this case, how to find a good starting vocabulary/protokenization results?\n2. The formulation and Figure 3 assume that the token frequency distribution is proportion to the sampling weight $\\alpha_i$, which further assumes that the corpus token frequency is somehow equally distributed across the corpus. Does that make sense? Can we empirically verify these assumptions? Do we need a very large corpus so that these assumptions hold true?\n3. How does the vocabulary size, size of the data and number of merges $T$ affect the time the attack requires to run?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Data Mixture Inference Attack: BPE Tokenizers Reveal Training Data Compositions\" presents a significant contribution to the field of machine learning by introducing a novel attack called \"data mixture inference.\" This attack aims to uncover the composition of pretraining data used in language models (LMs), an aspect often kept secret even when the model parameters are open-sourced. By leveraging information from byte-pair encoding (BPE) tokenizers, a common component of modern LMs, the authors successfully infer the proportions of different domains, languages, or code in the training data.\n\nSTRENGTHS:\n1. One of the key strengths of the paper is its demonstration of the attack's effectiveness. Through controlled experiments on known mixtures of natural languages, programming languages, and data sources, the authors show that their attack can recover mixture ratios with high precision, outperforming random guessing by several orders of magnitude. The attack's application to off-the-shelf tokenizers released with commercial LMs reveals new information about their training data composition, underscoring the practical implications of this work.\n\n2. The paper presents an efficient algorithm for solving the data mixture inference problem. Initially, the problem is formulated as a linear program (LP) with a large number of constraints (scaling with the cube of the vocabulary size, O(V^3)). However, the authors introduce several optimizations, such as truncating the merge list, efficiently storing pair counts, and using lazy constraint generation. These optimizations make the problem computationally tractable, allowing for practical application of the attack.\n\n3. The practical implications of this attack are noteworthy. It can potentially allow for the auditing of pretraining data for biases, reveal proprietary information about model construction, and enable targeted data poisoning attacks. \n\n4. The paper is well-written and organized, with the inclusion of an appendix with additional details and results further enhances the paper's clarity and accessibility.\n\nWEAKNESSES:\n1. The attack is specifically designed for BPE tokenizers, which, while widely used, are not the only tokenization method. The authors acknowledge this limitation and suggest that other tokenization algorithms might also leak information about training data, but the generalizability of the attack to other tokenization methods is not explored in this work.\n\n2. The attack's reliance on sampling and estimation of pair frequencies could lead to false positives or negatives, especially when dealing with low-resource languages or domains. The paper does not provide a detailed analysis of the attack's robustness to such sampling errors.\n\nQUESTIONS:\n1. The central idea of your paper relies on the fact that the pretraining data was used to train the tokenizer, with the proportions of available data determining the merge rules. However, what happens if a similar domain dataset is used to train the tokenizer with different proportions than those used during pretraining? For instance, if the tokenizer is trained with 40% coding, 40% English, and 20% non-English data, but the pretraining data proportions are 20% coding, 50% English, and 30% non-English, how would this affect the effectiveness and accuracy of your data mixture inference attack? Have you considered or tested such scenarios in your experiments?\n\n2. What are the implications of knowing the composition of the pretraining data? Specifically, what types of attacks could this knowledge facilitate, and what are the general implications of data mixture inference? Why is it important to answer the question of data mixture inference? If your paper addresses these points, please elaborate on how this knowledge could impact the security and integrity of language models. Because by just knowing the broad mixture categories and their weights, it doesn't allow for targeted data poisoning. But it seems it can be a stepping stone for building a sophisticated attacker. Would like to have your thoughts on the same?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}