Title: Tools for Verifying Neural Models' Training Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Scale Learning

ML Security

AI Governance
Abstract: It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a "Proof-of-Training-Data": any protocol that allows a model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of-Training-Data that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to a random seed used in training, and a method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point was included in training. We show experimentally that our verification procedures can catch a wide variety of attacks, including all known attacks from the Proof-of-Learning literature.

Tools for Verifying Neural Models' Training Data

Abstract

It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a 'Proof-of-Training-Data': any protocol that allows a model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of-Training-Data that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to a random seed used in training, and a method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point was included in training. We show experimentally that our verification procedures can catch a wide variety of attacks, including all known attacks from the Proof-of-Learning literature.

1 Introduction

How can we verify the capabilities of large machine learning models? Today, such claims are based on trust and reputation: customers and regulators believe that well-known companies building AI models wouldn't lie about the training data used in their models. However, as the ability to build new AI models proliferates, users need to trust an ever-larger array of model providers at their word, and regulators may increasingly face malicious AI developers who may lie to appear compliant with standards and regulations. Worse, countries developing militarily-significant AI systems may not trust each others' claims about these systems' capabilities, making it hard to coordinate on limits.

AI developers can enable greater trust by having a third party verify the developer's claims about their system, much as the iOS App Store checks apps for malicious code. Current black-box approaches to model auditing allow some probing of capabilities [Cen23], but these audits' utility is limited and a model's capabilities can be hidden [GTB22, GKVZ22]. An auditor can more effectively target their examination if they also know the model's training data, including the total quantity, inclusion of data likely to enable specific harmful capabilities (such as texts on cyber-exploit generation), and inclusion of safety-enhancing data (such as instruction-tuning [OWJ + 22]). However, if auditors rely on AI developers to self-report the data, developers could falsify their reports. This uncertainty limits the trust such audits can create.

In this work, we define the problem of Proof-of-Training-Data (PoTD): a protocol by which a thirdparty auditor (the 'Verifier') can verify which data was used by the developer (the 'Prover') to train a model. Our verification procedures assume that the Verifier can be given access to sensitive information and IP (e.g., training data, model weights) and is trusted to keep it secure; we leave the additional challenge of simultaneously preserving the confidentiality of the training data and model weights to future work. In principle, one could solve PoTD by cryptographically attesting to the

David Duvenaud

results of training on a dataset using delegated computation [CKV10]. However, in practice such delegation methods are impractically slow, forcing us to turn to heuristic verification approaches.

Inspired by the related literature on 'Proof-of-Learning' (PoL)[JYCC + 21], we propose that modeltrainers disclose a training transcript to the Verifier, including training data, training code, and intermediate checkpoints. In Section 4, we provide several verification strategies for a Verifier to confirm a training transcript's authenticity, including new methods that address all published attacks in the Proof-of-Learning literature. We demonstrate the practical effectiveness of our defenses via experiments on two language models (Section 6). Our methods can be run cheaply, adding as little as 1.3% of the original training run's compute. Further, we require no change to the training pipeline other than fixing the data ordering and initialization seeds, and storing the training process seeds for reproducibility. Still, like PoL, they sometimes require re-running a small fraction of training steps to produce strong guarantees.

The verification strategies we describe are not provably robust, but are intended as an opening proposal which we hope motivates further work in the ML security community to investigate new attacks and defenses that eventually build public confidence in the training data used to build advanced machine learning models.

2 Related Work

We build on [Sha23], which sketches a larger framework for verifying rules on large-scale ML training. It defines, but does not solve, the 'Proof-of-Training-Transcript' problem, a similar problem to Proof-of-Training-Data that additionally requires verifying hyperparameters.

Proof-of-Learning. [JYCC + 21] introduce the problem of Proof-of-Learning (PoL), in which a Verifier checks a Prover's ownership/copyright claim over a set of model weights by requiring the Prover to prove that they did at least as much computational work as was required to have trained the original model. The Prover's proof consists of reporting a sequence of valid weight checkpoints that led to the final weights. The Verifier then re-executes training between a few checkpoints and checks that the results are similar to confirm (or reject) the report's correctness. Unlike PoL, Proof-ofTraining-Data requires proving what exact data was used to produce a model. This is strictly harder: any valid PoTD protocol can serve as a solution to PoL, as compute cost can be derived from the size of the dataset. Furthermore, PoL only requires robustness to adversaries that use less computation than the original training run, whereas PoTD targets all computationally-feasible adversaries.

Several works have successfuly attacked the original PoL scheme; our proposed defenses defeat all published attacks. [ZLD + 22] and [KRCC22] provide methods for forging fake data that will (during retraining) interpolate between chosen checkpoints, thus allowing the production of fake transcripts. (See Section 4.3 for our mitigation.) [FJT + 22] exploit the fact that, due to the prohibitive cost of retraining, the Verifier can only retrain a tiny fraction of checkpoints. By falsifying only a modest fraction of checkpoints, a Prover can prevent their fake transcript being caught with high probability. (See Section 4.2 for our mitigation, which introduces a method for verifying checkpoints that is much cheaper than retraining, and can thus be applied to all checkpoints.)

Memorization during training. [ZIL + 21] introduce the notion of counterfactual memorization (the average difference in model performance with and without including a specific point in training) that is most similar to our own, and use it to investigate different training points' effects on final model performance. [FZ20] examine which datapoints are most strongly memorized during training by using influence functions, but they focus on the degree of memorization only at the end of training. [BPS + 23] show that per-datapoint memorization of text (as measured by top-1 recall) can be somewhat reliably predicted based on the degree of memorization earlier in training. [KGG + 22] analyze pointwise loss trajectories throughout training, but do not focus specifically on the phenomenon of overfitting to points in the training set.

3 Formal Problem Definition

In the Proof-of-Training-Data problem, a Prover trains an ML model and wants to prove to a Verifier that the resulting target model weights W ∗ are the result of training on data D ∗ . If a malicious Prover used training data that is against the Verifier's rules (e.g., terms of service, regulatory rules) then that

̸

Prover would prefer to hide D ∗ from the Verifier. To appear compliant, the Prover will instead lie and claim to the Verifier that they have used some alternative dataset D = D ∗ . However, the Prover will only risk this lie if they believe that with high probability they will not get caught (making them a 'covert adversary' [AL07]). The goal of a Proof-of-Training-Data protocol is to provide a series of Verifier tests that the Prover would pass with high probability if and only if they truthfully reported the true dataset that was used to yield the model W ∗ .

Let D ∈ X n be an ordered training dataset. Let M contain all the hyperparameters needed to reproduce the training process, including the choice of model, optimizer, loss function, random seeds, and possibly details of the software/hardware configuration to maximize reproducibility.

Definition 1. A valid Proof-of-Training-Data protocol consists of a Prover protocol P , Verifier protocol V , and witnessing template J that achieves the following. Given a dataset D ∗ and hyperparameters M ∗ , an honest Prover uses P to execute a training run and get ( W ∗ , J ∗ ) = P ( D ∗ , M ∗ , c 1 ) , where W ∗ ∈ R d is a final weight vector, J ∗ ∈ J is a witness to the computation, and c 1 ∼ C 1 is an irreducible source of noise. The Verifier must accept this true witness and resulting set of model weights with high probability: Pr c 1 ∼ C 1 ,c 2 ∼ C 2 [ V ( D ∗ , M ∗ , J ∗ , W ∗ , c 2 ) = 1] ≥ 1 -δ 1 , where δ 1 ≪ 1 / 2 and c 2 is the randomness used by the Verifier.

̸

Conversely, ∀ computationally-feasible probabilistic adversaries A which produce spoofs ( D,M,J ) = A ( D ∗ , M ∗ , J ∗ , W ∗ , c 3 ) where D = D ∗ and c 3 ∼ C 3 is the randomness used by the adversary, the Verifier must reject all such spoofs with high probability: Pr c 1 ∼ C 1 ,c 2 ∼ C 2 ,c 3 ∼ C 3 [ V ( D,M,J,W ∗ , c 2 ) = 0] ≥ 1 -δ 2 where δ 2 ≪ 1 / 2 .

Following the literature on the related Proof-of-Learning problem [JYCC + 21], we use as a witness the series of m model weight checkpoints J ∗ = W = ( W 0 , W 1 , . . . , W m -1 , W ∗ ) . Model weight checkpoints are already routinely saved throughout large training runs; we assume a checkpoint is saved after training on each k = n/m -datapoint segment. During verification, the Prover provides the Verifier with the training transcript T = { D,M, W} , which the Verifier will then test to check its truthfulness.

In practice, we cannot yet pursue provable robustness to all probabilistic adversaries A , due to the nascent state of the theoretical foundations of deep learning [FJT + 22]. Instead, as is done in the PoL literature, we approximate robustness to A by proposing a range of adversaries and then showing a protocol that defeats them. In particular, we will consider two major types of attacks (and their combinations):

̸

Non-Uniqueness Attack : The Prover forges a different D = D ∗ and M (and corresponding checkpoints W ) that would, if training was reexecuted, also lead to W ∗ . In general, it is easy to produce such transcripts (e.g., initialize training at W 0 = W ∗ and take no steps). In Section 4.3 we propose constraints on P that block such attacks.

Data Subtraction Attack : The Prover reports training on D , but secretly only trains on a subset D ∗ ⊊ D .

Data Addition Attack : The Prover reports training on D , but secretly trains on an additional set of data D ∗ ⊋ D .

Checkpoint Glue-ing Attack : The Prover reports a pair of successive checkpoints ( W i , W i +1 ) ⊂ W , but W i +1 was not produced from W i by a training procedure at all.

We address the latter three attacks in Section 6.

As a brute-force solution to Proof-of-Training-Data, the Verifier could simply re-execute the complete training process defined by T , and check that the result matches W ∗ . However, beyond technical complications 2 , doing so is far too computationally expensive to be done often; a government Verifier would need to be spending as much on compute for audits as every AI developer combined. Therefore any verification protocol V must also be efficient , costing much less than the original training run. Inevitably, such efficiency makes it near certain that the Verifier will fail to catch spoofs D = D ∗ if D only differs in a few data points; in practice, we prioritize catching spoofs which deviate on a

̸

substantial fraction of points in D ∗ . Though we do not restrict to a particular definition of dataset deviations, we list several possibilities relevant for different Verifier objectives in Appendix D.

4 Verification Strategies

We provide several complementary tools for detecting whether a transcript T is spoofed. Combined, these methods address many different types of attacks, including all current attacks from the PoL literature [ZLD + 22, FJT + 22].

4.1 Existing Tools from Proof-of-Learning

Our protocol will include several existing spoof-detection tools from the Proof-of-Learning literature [JYCC + 21], such as looking for outliers in the trajectory of validation loss throughout training, and plotting the segment-wise weight-change ∥ W i -W i -1 ∥ 2 between the checkpoints W . The most important of these existing tools is the segment-wise retraining protocol of [FJT + 22]. Let R ( W i -1 , Π i , c ; M ) be the model training operator that takes in a weight checkpoint W i -1 , updates it with a series of gradient steps based on training data sequence Π i (describing the order in which the Prover claims data points were used in training between checkpoints W i -1 and W i , which may be different from the order of the dataset D ∗ ), hyperparameters M , and hardware-noiserandomness c ∼ C , and then outputs the resulting weight checkpoint W i . Transcript segment i is ( ϵ, δ ) -reproducible if for the pair of checkpoints ( W i -1 , W i ) in W , the reproduction error (normalized by the overall segment displacement) is small:

The values ϵ and δ trade off false-positive vs. false-negative rates; see [JYCC + 21, FJT + 22] for discussion. The Verifier can use this retraining procedure as a ground-truth for verifying the faithfulness of a suspicious training segment. However, this test is computationally-intensive, and can thus only be done for a small subset of training segments. Our other verification strategies described in Sections 4.2 and 4.3 will be efficient enough to be executable on every training segment.

4.2 Memorization-Based Tests

The simplest way for a Prover to construct a spoofed transcript ending in W ∗ is to simply make up checkpoints rather than training on D ∗ , and hope that the Verifier lacks the budget to retrain a sufficient number of checkpoints to catch these spoofed checkpoints. To address this, we demonstrate a heuristic for catching spoofed checkpoints using a small amount of data, based on what is to the best of our knowledge a previously-undocumented phenomenon about local training data memorization.

Machine learning methods notoriously overfit to their training data D , relative to their validation data D v . We can quantify the degree of overfitting to a single data point d on a loss metric L : X × R | W | → R relative to a validation set D v via a simple memorization heuristic M :

Recall that Π i is the sequence of data points corresponding to the i th segment of the training run. One would expect that in checkpoints before data segment i , for data points d ∈ Π i , memorization M ( d, W j<i ) would in expectation be similar to the validation-set memorization; after data-segment i , one would expect to see higher degrees of overfitting and therefore M ( d, W j ≥ i ) would be substantially higher. We find evidence for this effect in experiments on GPT-2-Small [RWC + 19] and the Pythia suite [BSA + 23]). As shown in Figures 1 and 2, when a Prover reports the true training data, on average the greatest memorization occurs where Π i and W j = i match. We corroborate this finding with additional experiments on a range of models in Appendix G. The finding is even clearer if we look at jumps in memorization level, which we call the Memorization Delta ∆ M :

To test whether each reported checkpoint W i resulted from training on at least some of the segment training data Π i , a Verifier can compute a memorization plot like the one shown in Figure 1. Such

0

2

4

6

8

10

12

14

16

1

3

5

7

9

11

13

15

17

Data segment

Loss

0

2

4

6

8

10

12

14

16

1

3

5

7

9

11

13

15

17

Data segment

Memorization

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Memorization Delta

10

0

10

0

1

2

3

4

5

6

Memorization

×10

2

4

5

6

7

8

9

10

0

1

2

3

4

5

6

×10

2

4

2

0

2

4

6

×10

2

2

4

6

8

10

12

14

16

Data Segment

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

0

2

4

6

8

10

12

14

16

1

3

5

7

9

11

13

15

17

Data segment

GPT-2 2 Epochs

0

2

4

6

8

10

12

14

16

1

3

5

7

9

11

13

15

17

Data segment

0.5

0.0

0.5

1.0

1.5

×10

2

2

1

0

1

2

3

4

5

×10

2

1

0

1

2

3

4

5

6

×10

2

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0.25

0.50

×10

3

plots can be computed more efficiently by sampling only a small fraction α of the training data Π , and by plotting only a few checkpoints W i -β , . . . , W i + β for each segment Π i .

We can further harness this memorization phenomenon to test whether on segment i , rather than training on the full claimed data sequence Π i and yielding W i , the Prover secretly skipped training on at least a κ -fraction of the points in Π i and yielded W ′ i . Consider the odds that, for d ∼ Π i , ∆ M ( d, W i ) happens to fall in the bottom p -probability quantile of the validation set D v 's ∆ M values on W i :

̸

where I is the indicator function, PBQ stands for 'Point is Below Quantile', and FBQ stands for 'Fraction Below Quantile'. We can see in Figure 3 that, as expected, when the points in Π i are all included in training, FBQ (Π i , p, W i ) is small compared to FBQ (Π j = i , p, W i ) ). If many points were secretly excluded, as in W ′ i , we should expect that FBQ (Π i , p, W ′ i ) should be higher and closer to FBQ ( D v , p, W ′ i ) , where D v is the validation set. If the Prover secretly excluded a greater than κ -fraction of data points in Π i thus yielding W ′ i , then we should expect that:

(see derivation in Appendix B). Rearranging terms, we get

λ (Π i , p, W ′ i ) can be estimated using a small fraction of training and validation datapoints, and can serve as an upper-bound estimate on κ , the fraction of Π i secretly excluded from training W ′ i . 3 In

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Frac. of points above 50th percentile

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Frac. of points below 10th percentile

2

1

0

1

2

Memorization Delta

0.00

0.25

0.50

0.75

1.00

1.25

Density

Checkpoint #1

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #9

train set

valid. set

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

10

3

10

2

10

1

Section 6 we show that this heuristic can detect even small data subtractions in practice, and in Appendix H.2 we show the test's effectiveness across a range of percentiles p and segment-lengths k .

We also observe that M gradually decreases over time from an initial peak immediately after the point's training segment. This echoes the many findings on 'forgetting' in deep learning [TSC + 18]. We show in Section 6 how this can be used to catch gluing attacks.

4.3 Fixing the Initialization and Data Order

As mentioned in Section 3, a Proof-of-Training-Data protocol needs to defend against non-uniquness attacks, by making it difficult for a malicious Prover to produce a second transcript with D = D ∗ that, if training was legitimately executed, would also end in W ∗ . There are two well-known types of attacks the Prover might use to efficiently produce such spoofs:

̸

Initialization attacks : An attacker can choose a 'random' initialization that places W 0 in a convenient position, such as close to the target W ∗ . Even if the Verifier uses statistical checks to confirm that the initialization appears random, these are sufficiently loose that an adversary can still exploit the choice of initialization [ZLD + 22].

Synthetic data/data reordering attacks : Given the current weight vector W i , an attacker can synthesize a batch of training datapoints such that the resulting gradient update moves in a direction of the attacker's choosing, such as towards W ∗ . This could be done through the addition of adversarial noise to existing data points [ZLD + 22], generating a new dataset [TJSP22], or by carefully reordering existing data points in a 'reordering attack' [SSK + 21].

We propose methods for preventing both of these attacks by forcing the Prover to use a certifiedrandom weight initialization, and a certified-random data ordering. The randomized data ordering guarantees that the adversarial Prover cannot construct synthetic datapoints that induce a particular gradient, because it does not know the corresponding weights W at the time of choosing the datapoints D . Given a fixed data ordering, we discuss in Appendix E why it may be super-polynomially hard to find a certified-random weight initialization that, when fully trained, results in a particular W ∗ .

The Verifier can produce this guaranteed-random initialization and data order by requiring the Prover to use a particular random seed s , constructed as a function of the dataset D itself . This produces the initialization W 0 = G r ( s ) ∈ X n and data ordering S = G p ( s ) using a publicly known pseudorandom generators G r and G p . 4 5 The Prover can also construct a verifiable validation subset D v by holding

out the last n v data-points in the permutation S from training. The Prover constructs s as follows. Assume that the dataset D has some initial ordering. Let H be a publicly-known cryptographic hash function. We model H as a random oracle, so that when composed with G r or G p , the result is polynomial-time indistinguishable from a random oracle. 6 This means that if a Prover wants to find two different seeds s 1 , s 2 that result in similar initializations W 0;1 , W 0;2 or two similar permutations S 1 , S 2 , they can find these by no more efficient method than guessing-and-checking. For large d and n , finding two nontrivially-related random generations takes exponential time. We construct the dataset-dependent random seed s as

where { d 1 , . . . , d a } = D , ◦ is the concatenation operator, and s rand is a Prover-chosen 32-bit random number to allow the Prover to run multiple experiments with different seeds. 7 A Verifier given access to D (or only even just the hashes of D ) can later rederive the above seed and, using the pseudorandom generators, check that it produces the reported W 0 and S .

The important element of this scheme is that given an initial dataset D ∗ and resulting data order S , modifying even a single bit of a single data point in D ∗ to yield a second D will result in a completely different data order S ′ that appears random relative to S . Thus, if we can statistically check that a sequence of checkpoints W matches a data order S ∗ and dataset D ∗ better than a random ordering, this implies that D ∗ is the only efficiently-discoverable dataset that, when truthfully trained 8 , would result in the checkpoints W and final weights W ∗ . We provide this statistical test in Appendix C.

This same approach can be extended to the batch-online setting, where a Prover gets a sequence of datasets D ∗ 1 , D ∗ 2 , . . . and trains on each before seeing the next. The Prover simply constructs a new seed s ( D ∗ i , s rand ) for each dataset D ∗ i , and continues training using the resulting data ordering. This works so long as each D ∗ i is large enough for a particular data-ordering to not be brute-forceable.

4.4 Putting It All Together

In Appendix A we sketch a complete protocol for combining these defenses complementarily to detect all of the attacks discussed in Section 6. The overall computational cost for the Verifier is O ( n ) training data-point hashes, O ( αn ) model inferences for computing losses, and O ( | Q | n ) gradient computations for retraining transcript segments (where | Q | depends on hyperparameters that can be adjusted according on the Verifier's compute budget). Importantly, the Verifier's cost grows no worse than linearly with the cost of the original training run. If we run our tests using an α = 0 . 01 fraction of the points in each segment as done in our experiments below, the verification cost of computing our new tests in Sections 4.2 and 4.3 totals just 1.3% of the original cost of training, assuming inference is 3 × cheaper than training.

5 Experimental Setup

Our main experiments are run on GPT-2 [RWC + 19] with 124M parameters and trained on the OpenWebText dataset [GCPT19]. We use a batch size of 491,520 tokens and train for 18,000 steps ( ∼ 8.8B tokens), which is just under 1 epoch of training, saving a checkpoint every 1000 steps. See Appendix F for additional details. The data addition attack experiments in Section 6 further use the Github component of the Pile dataset [GBB + 20] as a proxy for a Prover including additional data that is different from reported data. In addition to training our own models, we also evaluate Pythia checkpoints [BSA + 23] published by EleutherAI, as they publish the exact data order used to train their models. We chose the 70M, 410M, and 1B-sized Pythia models trained on the Pile dataset with deduplication applied. All experiments were done using 4 NVIDIA A40 GPUs.

6 Empirical Attacks and Defenses

Below, we show that our methods address the last three attacks (Checkpoint Glue-ing, Data Subtraction, and Data Addition). We omit the synthetic initialization and synthetic data attacks of [FJT + 22, ZLD + 22] as we addressed those in Section 4.3. All plots are from experiments using GPT-2; we include additional experiments in Appendix H. We do not claim that the attacks studied here are exhaustive, but provide them as a starting point to motivate future work.

Checkpoint Glue-ing Attack A known attack against Proof-of-Learning, which also applies to PoTD, is to 'glue' two training runs W A and W B together and report a combined sequence of checkpoints W = ( W A 0 , . . . , W A i , W B j ≫ 0 , . . . , W B final ) . The resulting model W B final can be trained on undisclosed data prior to segment j , with the Prover never reporting this data to the Verifier. As highlighted by [JYCC + 21], the size of the glued segment ∥ W B j -W A i ∥ 2 will generally appear as an outlier in weight-space. We demonstrate this phenomenon in Figure 4. Following [JYCC + 21], a Verifier could then check such suspicious segments via retraining. We demonstrate a second verification option using inference instead of training: the Verifier can check whether the checkpoint W B j has memorized not only the most recent data Π i , but also the preceding data segments Π i -1 , Π i -2 , . . . The absence of long-term memorization is visible in the memorization heatmap in Figure 4.

1

3

5

7

9

11

13

15

17

i

)

0

50

100

150

200

250

300

350

400

||

Wi

Wi

1||

0

2

4

6

8

10

12

14

16

18

1

3

5

7

9

11

13

15

17

Data segment

Memorization

0

5

10

15

0

1

2

3

4

5

6

Memorization

×10

2

0

1

2

3

4

5

6

×10

2

1

2

3

4

5

6

7

8

9

Data Segment

To avoid the spike in weight-space shown in Figure 5 when jumping from W A i to W B j , the attacker can break up the large weight-space jump into smaller jumps by artificially constructing intermediate checkpoints aW B j +(1 -a ) W A i for several values of a . However, these interpolated checkpoints fail our memorization tests, as they are artificial and not the result of actual training (Figure 5).

Data Subtraction Attack In a data subtraction attack, a Prover claims the model has been trained on more points than it truly has. Detecting data subtraction attacks could enable a Verifier to detect overclaiming by model providers, including claiming to have included safety-related data when they secretly did not. Subtraction can also be used to hide data addition attacks, as combining the two attacks would mean the segment was still trained on the correct number of datapoints, thus suppressing the weight-change-plot signature used to catch data addition (as in Figure 7). We demonstrate the effectiveness of an efficient memorization-based approach for detecting subtraction, described in Section 4.2. Leveraging the subtraction-upper-bound test from Equation 7, we see in Figure 6 that the upper-bound heuristic λ (Π , p, W i ) is surprisingly tight, consistently differentiating between no-subtraction segments and even small subtraction attacks. Still, even if λ (Π i , p, W ) > z for some large z ≫ 0 , this is only an upper bound on the quantity of data subtraction, and does not prove that a z -fraction of points were subtracted. The Verifier can instead use this test as an indicator to flag segments for retraining, which would confirm a subtraction attack. (That retraining would

0

5

10

15

3.0

3.2

3.4

3.6

3.8

4.0

Validation Loss

Original Run

Interpolated Run

0

2

4

6

8

10

12

14

16

18

1

3

5

7

9

11

13

15

17

Data segment

Memorization

6

5

4

3

2

1

0

1

2

×10

2

̸

5

10

15

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Subtraction Test with

p

=0.1

Full Data

5% subtraction

50% subtraction

95% subtraction

5

10

15

0.0

0.2

0.4

0.6

0.8

1.0

Subtraction Test with

p

=0.2

Full Data

5% subtraction

50% subtraction

95% subtraction

2

1

0

1

2

Memorization Delta

0.0

0.5

1.0

1.5

2.0

Density

Checkpoint #1

train set

valid. set

0.2

0.1

0.0

0.1

0.2

Memorization Delta

0

5

10

15

20

Density

Checkpoint #9

train set

valid. set

result in a different weight vector can be inferred from the plot of the 50%-addition attack in Figure 7). Appendix H.2 explores the test's performance on the suite of Pythia models.

Data Addition Attack Addition attacks occur when a Prover in addition to training on the declared dataset D and data sequence Π , trains on additional data D ′ without reporting it to the Verifier. This can be used to secretly enhance a model's capabilities, install backdoors, or train on restricted data. This attack cannot be detected using memorization analysis (Figure 7), because the Verifier does not know and cannot test points d ′ ∈ D ′ . However, Figure 7 also shows that significant amount of data addition, or data addition from a different distribution ,causes a larger weight-displacement in that segment, which can be detected by looking at the segment's magnitude. If a Prover tries to hide this by deleting an equally-sized subset of points from D , that can be detected as a subtraction attack. We leave exploration of other attacks and defenses, like changing learning rates, to future work.

This raises the problem of how to estimate the 'expected' segment length, which may require retraining on a chosen subset of segments, and interpolating to estimate other segments' lengths. If, to hide the non-smoothness of adding extra data to a single segment, the Prover adds data uniformly throughout a large fraction of the training run, then choosing even a small number of segments randomly should be sufficient to catch at least one offending segment with high probability. Unfortunately, these defenses would not detect an attacker that adds a modest amount of data within a small number of segments, that cannot be detected by the segment-magnitude or data subtraction defenses.

0

2

4

6

8

10

12

14

16

1

3

5

7

9

11

13

15

17

Data segment

Memorization

1

3

5

7

9

11

13

15

17

i

)

20

40

60

80

100

120

140

||

Wi

Wi

1||

Add 50% (OWT)

Add 5% (Github)

Baseline (OWT)

0

9

17

0.0

0.2

0.4

0.6

0.8

Reproduction Error

Add 50% (OWT)

Add 5% (Github)

Baseline (OWT)

0

1

2

3

4

5

×10

2

7 Discussion and Limitations

This work contributes to an emerging societal effort to develop practical and robust tools for accountability in the large-scale development of AI models. The statistical tests we introduce are best taken as an opening proposal. Future work could propose clever new attacks that break this protocol, or better yet, create new defenses that efficiently detect more, and subtler, attacks and enable trustworthy verification of ML models' training data.

Experimental Limitations This work provides suggestive evidence for the local-memorization phenomenon, but further study is needed across additional modalities, architectures, and training recipes in order to determine its broad applicability. Encouragingly, we find in Appendix G that local-memorization gets even stronger as models get larger, though memorization appears weaker near the end of training as the learning rate shrinks. The paper's experiments only include language models, in part because they are a current priority for audits. The memorization tests used may need to be adjusted for models trained with less data on many epochs, such as image models [YZS + 22].

Attacks Our Protocol Does Not Catch There are several remaining directions for attacks. The attacks explored above can be composed in new ways, and it may be possible for compositions of attacks to undermine the defenses that would otherwise detect each attack individually. The method also does not address small-scale data additions, and thus cannot yet detect copyright violations or spot inserted backdoors [XWL + 21]. It also cannot detect attacks based on small-norm modifications to the weights, which could be used to insert backdoors [BISZ + 22]. Finally, attacks could be masked with cleverly chosen hyperparameters, such as by using a temporary lower-than-reported learning rate to shrink large changes in W . Exploring whether such attacks are feasible without degrading learning performance - and identifying defenses - is an interesting direction for future work.

Applicability to Different Training Procedures We attempted to make our procedure as agnostic as possible to the details of the training procedure, and believe it will be compatible with most training procedures for large models in use today. However, our protocol does not apply to online or reinforcement learning, or to schemes that require multiple models to be co-trained [GPAM + 20], as the data is unknown in advance. This means the uniqueness defense cannot be applied (Section 4.3). Finding methods for defending against non-uniqueness attacks even in the online setting is a valuable direction for future work.

Maintaining Privacy and Confidentiality One significant challenge to using this protocol in practice is that, just like with PoL [JYCC + 21], it requires that the Prover disclose confidential information to the Verifier, including training data, model weights, and code. In principle, the Prover may only need to disclose hashes of the data and weights to the Verifier, with the matching full data and weights only ever supplied on the secure cluster during verification.

References

[AHS22] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836 , 2022.

[AL07] Yonatan Aumann and Yehuda Lindell. Security against covert adversaries: Efficient protocols for realistic adversaries. In Theory of Cryptography: 4th Theory of Cryptography Conference, TCC 2007, Amsterdam, The Netherlands, February 21-24, 2007. Proceedings 4 , pages 137-156. Springer, 2007.

[BISZ + 22] Mikel Bober-Irizar, Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot. Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07840 , 2022.

[BPS + 23] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158 , 2023.

[BSA + 23] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.

[Cen23] Alignment Research Center. Update on ARC's recent eval efforts, March 2023.

[CKV10] Kai-Min Chung, Yael Kalai, and Salil Vadhan. Improved delegation of computation using fully homomorphic encryption. In Advances in Cryptology-CRYPTO 2010: 30th Annual Cryptology Conference, Santa Barbara, CA, USA, August 15-19, 2010. Proceedings 30 , pages 483-501. Springer, 2010.

[FDRC20] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259-3269. PMLR, 2020.

[FJT + 22] Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, and Nicolas Papernot. On the fundamental limits of formally (dis) proving robustness in proof-of-learning. arXiv preprint arXiv:2208.03567 , 2022.

[FZ20] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2881-2891. Curran Associates, Inc., 2020.

[GBB + 20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 , 2020.

[GCPT19] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwebtext corpus, 2019.

[GKVZ22] Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS) , pages 931-942. IEEE, 2022.

[GPAM + 20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

[GTB22] Wei Guo, Benedetta Tondi, and Mauro Barni. An overview of backdoor attacks against deep neural networks and possible defences. IEEE Open Journal of Signal Processing , 2022.

[JYCC + 21] Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 1039-1056. IEEE, 2021.

[KGG + 22] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv preprint arXiv:2202.09931 , 2022.

[KRCC22] Zhifeng Kong, Amrita Roy Chowdhury, and Kamalika Chaudhuri. Forgeability and membership inference attacks. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security , pages 25-31, 2022.

[OWJ + 22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

[RWC + 19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv preprint arXiv:1901.11196, 2019.

[Sha23] Yonadav Shavit. What does it take to catch a Chinchilla? Verifying rules on large-scale neural network training via compute monitoring. arXiv preprint arXiv:2303.11341 , 2023.

[SSK + 21] Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems , 34:18021-18032, 2021.

[TJSP22] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22) , pages 4007-4022, 2022.

[TSC + 18] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 , 2018.

[XWL + 21] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 103-120. IEEE, 2021.

[YZS + 22] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 , 2022.

[ZIL + 21] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938 , 2021.

[ZLD + 22] Rui Zhang, Jian Liu, Yuan Ding, Zhibo Wang, Qingbiao Wu, and Kui Ren. 'adversarial examples' for proof-of-learning. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 1408-1422, 2022.

A Combined Verification Protocol

We can unify the defenses of Section 4 into a combined defense protocol, which catches a wide swath of attacks, including all current attacks on from the Proof-of-Learning literature [FJT + 22, ZLD + 22].

A Prover gives the Verifier a transcript T = { D,M, W} and a final weight vector W ∗ . The verifier proceeds to verify whether T is a valid training transcript through the following checks:

Check that W ends in the claimed final weights W ∗ .

Given the dataset D , hash it to yield the seed s as in Section 4.3, and use that seed compute the resulting data order Π and validation subset D v . (Alternatively, these hashes can be provided by the Prover, and only verified when each point is needed for the protocol.)

Check that W 0 matches G r ( s ) . If this fails, reject the transcript.

Create an empty list Q to store all suspicious-looking segments to retrain. For each segment W i , Π i , include it in the list Q to retrain if it fails any of the following checks:

(a) Randomly select an α fraction (e.g., 1% of k ) of points Π i,α from Π i . For each such point d ∼ Π i,α , compute the losses on W i and W i -1 , shorthanded as sets L Π i ,i and L Π i ,i -1 . Similarly, pick an α fraction 9 of points from the validation set D v and compute these points' losses on W i , shorthanded as L D v ,i . (The validation loss on W i -1 , L D v ,i -1 , should've already been computed when looping on the previous segment.) Also, randomly select an αk subset of data points D t from across all training segments Π , and compute these points' losses on W i , L D t ,i .

If the Verifier wants to plot complete memorization plots, for example as a sanity check or to use in checking for a Glue-ing attack as described in Section 6, they can also compute the losses on 2 β nearby weight checkpoints W i -β , . . . , W i + β -1 . However, this is not part of the core protocol, and will not be counted in its sample complexity.

(b) Compare the values in L Π i ,i and L D t ,i using the one-sided binomial hypothesis test described in Appendix C, to check that the model used the correct data ordering. If the test cannot reject the null hypothesis that Π i does not match W i better than randomlyselected training points D t , then run the test on an even larger fraction of data points from Π i and D to double-check the statistical significance of the null. If this still fails, it suggests that the transcript may have been spoofed. Investigate further (such as by reexecuting training on this segment) and/or reject the transcript.

(c) Using L Π i ,i , L Π i ,i -1 , L D v ,i , and L D v ,i -1 , calculate a data subtraction upper bound heuristic as in Section 4.2. If this heuristic fails to bound the data subtraction to below an acceptable small limit (which depends on how strongly the Verifier wants to catch even small subtraction attacks, and on the Verifier's retraining budget), add this segment to Q .

(d) Using {L D v ,j | j ∈ 0 , . . . , m } , compute the mean validation set loss across time, and check that it is smooth at W i and doesn't increase or decrease discontinuously. If it does, add this segment to Q . (The tolerance should depend on the retraining budget.)

(e) Compute the distance ∥ W i -W i -1 ∥ from the previous checkpoint. Check that this delta is similar to nearby segments' weight-space deltas. If it is not, add this segment to Q . (The tolerance should depend on the retraining budget. We leave the question of how best to set this threshold, and of σ below, to minimize false negatives while avoiding reducing false positives that increase retraining costs, to future work.)

Randomly sample σ additional data segments from throughout training, and add them to Q . These additional segments are important to establish baseline estimates of segments' weight-space deltas across training, to ensure that there were no systematic data addition attacks at every segment. (Illegal data additions in every segment would shift the entire weight-change delta magnitude plot, thus suppressing anomalies in any one segment).

For each segment in the list Q , execute retraining and verify that the resulting weights ˆ W i are within an ϵ -ball of the original reported weights W i reported in the transcript.

If any values in the re-trained weights fail to come within the tolerance ϵ , that is significant evidence that the transcript has been spoofed, and warrants further investigation. For example, the segment can be retrained more times, to confirm that the weight-space variance across retraining results ˆ W (1) , ˆ W (2) , . . . is sufficiently smaller than ϵ such that the reported W i is a clear outlier.

If all these tests pass, accept the transcript.

A.1 Complexity

The time costs of training, borne by the the Prover are:

h ×| D | , where h is the cost of a hash, for generating the initial random seed.

s × n , where s is the cost of a single gradient computation, and n is the number of training data points.

In comparison, the time costs to the Verifier (assuming the transcript is accepted) are:

h ×| D | hashes for verifying the initial weights.

(2 + 1 + 1) × α × s 3 × n operations for computing the loss of an α fraction of datapoints in Π i on W i and W i -1 , and another 2 α fraction of points in D t and D v . We also assume that computing the loss requires 1 3 the number of operations as computing a gradient update, which is the standard ratio of inference vs. training when using backpropagation.

s × n ×| Q | /m operations for retraining, where m is the total number of checkpoints in the training run.

B Subtraction Upper-Bound Heuristic Derivation

Let W i be the checkpoint obtained by training on the full claimed data sequence Π i , and let W ′ i be the checkpoint obtained by excluding κ -fraction of the points in Π i . Our goal is to estimate FBQ (Π i , p, W ′ i ) , the fraction of points in Π i that fall in the bottom p -probability quantile of D v 's ∆ M values on W ′ i .

Let's start with the distribution of ∆ M ( d, W i ) , where d ∼ Π i . We make the following two assumptions:

The subtracted κ -fraction of points are sampled randomly from Π i .

The p -th percentile of the ∆ M values of D v 's ∆ M values on W i is the same as on W ′ i .

Firstly, since κ -fraction of points were subtracted, we know that those subtracted points are distributed like the validation set (assumption 1). Therefore, the fraction of points in Π i that lie in our quantile of interest that were subtracted is κ FBQ ( D v , p, W ′ i ) .

Next, we must consider the points that were not subtracted but lie in the quantile. Using assumption 2, we know that (1 -κ ) fraction of points that were below the quantile remain in the quantile. Therefore, the fraction of points in Π i that lie in our quantile of interest that were not subtracted is (1 -κ ) FBQ (Π i , p, W i ) .

Putting it all together, we have

There are several things to note about this derivation. First of all, regardless of whether assumption 2 is correct, the second term of Equation 9 is ≥ 0 and therefore, λ is still an upper-bound. In practice, the actual fraction of subtracted points that lie in the quantile might not be exactly equal to FBQ ( D v , p, W ′ i ) = p . Therefore, if (1 -κ ) FBQ (Π i , p, W i ) is very small and close to zero, the resulting upper-bound can be smaller than κ , which is the case in Figure 6.

Lastly, when computing FBQ values, we can use the unnormalized M values to compute ∆ M as such:

This does not affect the FBQ values, since the normalization term E d ′ ∈ D v [ L ( d ′ , W )] does not depend on d and therefore and has no effect when evaluating ∆ M ( d ′ , W i ) > ∆ M ( d, W i ) . Removing the normalization term could represent a 2x efficiency gain. We kept the normalization term in the main paper because it may still be important in other cases for preserving the meaning of the Memorization Delta ∆ M , which is used to produce memorization-charts. Without the normalization term, it becomes hard to compare ∆ M across segments, especially in the earlier stages of training when the changes in loss between checkpoints vary quite a bit across segments.

C Data Order Statistical Test

We want a statistical test that will tell the Verifier whether, for a given training dataset D and data ordering S , which together yield a data sequence Π , and for a given weight checkpoint W i , the data segment sequence Π i ∈ X k explains the memorization pattern of W i better than a random data order/sequence Π ′ i (which we assume is drawn randomly from D ). In particular, based on results from Section 4.2, we know that datapoints from the most recent training segment d ∈ Π i tend to have higher memorization delta values ∆ M than the average point d ′ ∈ D from the overall training distribution D . Conversely, points from Π ′ i would have no reliably greater ∆ M than the rest of D .

We will use the following test, where Π ? = Π ′ i is the null hypothesis and the alternative hypothesis is Π ? = Π i . Let z = median d ∈ D (∆ M ( d, W i )) , estimated via a small number of samples from D . For Pick n t datapoints from data sequence Π ? , and for each data point d ∈ Π ? , check if it's > z . Under the null hypothesis, the probability that each point passes this check is 0 . 5 . Let the test statistic be t :

where I is the indicator function. The value of t (Π ′ i ) , the statistic under the null hypothesis, is distributed as a binomial with biased coin probability c = 1 / 2 and n t samples. However, we expect that t (Π i ) is a binomial with a larger c . To compute our confidence that W i was trained using the data order Π i , we can use a one-sided binomial hypothesis test, computing a p -value as 1 -CDF binomial ( c = 1 / 2 , n t , k < t ) where k is the value up to which to calculate the CDF. This statistic can be computed jointly across all checkpoints (requiring relatively few samples per checkpoint) to prove that the overall data ordering matches the one defined in Section 4.3.

Note that this test is similar to the 'subtraction upper bound' heuristic from Section 4.2, with the key difference being that in this test we compare against the distribution of all training points D (since the counterfactual is a randomly selected subset of training data), whereas the subtraction test compares against points from the validation set D v (since the counterfactual is that the points are never included in training). As an additional note, this same test can be generalized by replacing the median with a quantile, which may improve sample efficiency depending on the shape of the ∆ M distribution on Π i vs. Π ′ i .

D Verifier Objectives Table

E Hardness of Spoofing a Weight Initialization

To recap, by requiring that the Prover initialize a model's weights at a specific value in highdimensional space W 0 ∈ R d drawn from a pseudorandom vector generator G r , we seek to disallow a class of spoofing attacks based on the Prover hand-picking an initial weight vector ˆ W 0 that will

after training end up close to W f , for example by picking an initialization that is already close to W f (Attack 2 in [ZLD + 22]).

The simplest setting in which defense is impossible, and the Prover can reliably find a random initialization that will converge to a given W f , is in realizable linear models (models with only a single linear layer). Since their loss function is strongly convex, any initialization will converge to a neighborhood of the same final value W f , making it straightforward to construct tweaked datasets with certified-random initializations that result in approximately the same final model. Another counterexample occurs when datasets have a single degenerate solution: it is possible to construct a 2-layer neural network with training data covering the input space and where all the labels are 0 , such that the model always converges to a weight vector of all 0 s, independent of initialization. We will focus our discussion on the usual case of multi-layer NNs with non-degenerate solutions, as described below.

Below, we will sketch an informal argument that for some radius r , for a fixed training data sequence Π , the probability that a training run initialized at a pseudorandomly-generated 10 weight vector W 0 = G r ( s ) ends in a final weight vector W f that is within distance r of a particular target vector A , is less than some small value δ < ˜ o (1 /poly ( d )) , where d is the dimension of the neural network. This means that a Prover would need to sample a super-polynomial (in d ) number of random seeds to find one that would, via training on Π , result in a fully-valid training transcript that ends close to the weight vector W f from a previous training run with a different initialization, and therefore that it is exponentially hard to violate the 'uniqueness' property from Section 3 if the Prover uses a certified random initialization.

To understand whether this is the case, we can examine the counterfactual claim: that independent of weight initialization, all NNs tend to converge to a small (polynomial) number of modes in weight space. This is indeed the case with linear regression: regardless of the initialization, given sufficient full-rank data all linear model training runs will converge to a neighborhood of the same loss minimum in weight-space. If this were also true for neural networks, then even a small number of randomly-sampled weight initializations would likely yield at least one weight initialization that, after training, converged to a mode close to the target A (assuming A is close to at least one mode, which is the case when A is the outcome of a previous training run W f ). Yet, empirically, many works have found that large NNs converge to many different modes [AHS22, FDRC20].

The many modes of the NN loss landscape can be understood through permutation symmetries [AHS22]. Neural networks are equivariant ('equivariant' means that a function changes symmetrically under a group action) under specific permutations of their matrices' columns and rows. Nearly all neural networks have the following permutation symmetries: given a single hidden layer M 1 σ ( M 2 ( x )) where M 1 ∈ R a × b , M 2 ∈ R b × c and σ : R b → R b is a nonlinearity, and given any permutation matrix F ∈ R b × b (such that FZ permutes the rows of Z ), then by simple algebra M 1 F T σ ( FM 2 x ) = M 1 σ ( M 2 x ) for all x . This means that for any set of successive NN matrices M 1 , M 2 , there are at least b ! possible permutations with identical input output behavior. For a neural network W with k nonlinear layers and hidden dimension of each layer b , there could be k -1 different permutation matrices F 1 , F 2 , . . . , and we denote to the operation of permuting the flattened weight vector W using a particular value of these F s as P : R d → R d . Each P is drawn from the overall set of valid permutations for a particular architecture P ∈ P ( M ) , and we know that ∥ P ( M ) ∥ = Ω ( 2 kb log b ) .

A second important property is that gradient descent is itself equivariant under the described permutations. Let R be the training operator, such that W f = R ( W 0 , Π) is the result of training initial weights W 0 on a data sequence Π . 11 Then it is true that ∀ P ∈ P ( M ) ,

where W p f is the result of training on the permuted initialization. This is simply a consequence of the fact that the gradient operator commutes with any constant matrix (including the permutation matrix), and that the training process R is comprised of repeated calls to the gradient operator, additions, and scalar multiplications (both of which also commute with the permutation matrix). 12

Now, assume that the initialization function G r is radially symmetric (as is the case with all common initialization schemes, e.g., those based on Gaussians), and therefore the probability that the initialization will start at W 0 and P ( W 0 ) is the same for all P ∈ P . Then the probability that the post-training final weights reach W f or P ( W f ) is also the same. If we knew that Pr W 0 ∼ G r ( s ) ( ∥ W f -P ( W f ) ∥ > 2 r ) > 1 -δ for some r and small δ , then this derivation would tell us that there are many different weight-space modes into which training could converge, each of which is far apart from the others. (For convenience, let's refer to the number of such far-apart permuted modes as k .)

Again, our goal is to show that a random initialization is unlikely to converge after training to within a neighborhood around some vector A . Assume that B is one of these modes, and ∥ A -B ∥ < r . 13 According to the assumption from the previous paragraph on the distance between post-training modes, for any second mode C , we know that ∥ C -B ∥ > 2 r with high probability. By the triangle inequality, we know that:

Therefore there is some minimum distance ∥ A -C ∥ > r between the target A and all other k disjoint modes (each associated with a permutation) of the post-training weight distribution. If the number of such far-apart permutations k is superpolynomial, then no polynomial number of weight initialization samples will result in a final model close to A .

However, this argument is predicated on a sometimes-invalid assumption: that there are superpolynomially-many permutations k = ω ( poly ( d )) of W f , each at least a distance 2 r from each other. In the case of the counterexample from the beginning, where all initializations converge after training to the weight vector of all 0 s, all such permutations are in fact equal, and therefore there is no such distance r . Instead, one may need to make an assumption about the non-degeneracy of the distribution of final weight vectors W f , such that permutations of these weight vectors are far apart from each other. We leave analysis of which assumptions fulfill this property as future work. Note that for any specific training transcript which includes a specific W f , the distribution of distances of permutations of W f can be estimated empirically by manually permuting W f 's matrices.

F Experiment Details

For the GPT-2 Experiments we use a cosine learning rate schedule that decays by a factor of 10x by the end of training, with a linear warmup of 2000 steps to a peak learning rate of 0.0006. For the Pythia evaluation experiments, we choose checkpoints from 3 contiguous blocks out of 144 checkpoints: early (first 19 checkpoints), mid (checkpoints at step 62000 to 80000), and late (last 19 checkpoints).

G More memorization plots

In the following subsections, we plot memorization M , fraction of points with ∆ M above the median, and fraction of points with ∆ M below the 10th percentile. For GPT-2, we use 100% of the data to generate Figures 1, 2, and 3, while for Pythia, we use 10% of the data to generate Figure 2. In this section, we show results for smaller sampling rates to highlight that with 1%, or sometimes even 0.1% of the original data, we can still observe the memorization effect.

From Pythia 70M results (Subsections G.1, G.2, and G.3) we can see that as training progresses, the memorization effect becomes less pronounced, such that with a smaller data sampling rate, less of the diagonal get highlighted (Figures 11, 18, 25), and the histograms are closely overlapping (Figure 32) for the last 18 checkpoints. At the same time, we observe that as the model size increases the memorization effect becomes clearer, even with 0.1% data sampling rate. In fact, for the 1Bparameter Pythia model, the memorization effect is still clear for the last few checkpoints (Figures 14, 21, 28, and 35) unlike the 70M-parameter case.

G.1 Memorization

0

2

4

6

8 10 12 14 16

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 1.0

0

2

4

6

8 10 12 14 16

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

0

2

4

6

8 10 12 14 16

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

0

2

4

6

8 10 12 14 16

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0

1

2

3

4

5

6

Memorization

×10

2

0

2

4

6

Memorization

×10

2

0

2

4

6

8

Memorization

×10

2

0.025

0.000

0.025

0.050

0.075

0.100

0.125

Memorization

GPT-2

120M

0

2

4

6

8

10 12 14 16 18

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

0

2

4

6

8

10 12 14 16 18

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

0

2

4

6

8

10 12 14 16 18

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Memorization

×10

2

2

1

0

1

2

Memorization

×10

2

8

6

4

2

0

2

4

Memorization

×10

2

Pythia

70M

Early

62 64 66 68 70 72 74 76 78 80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

62 64 66 68 70 72 74 76 78 80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

62 64 66 68 70 72 74 76 78 80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.0

0.5

1.0

1.5

2.0

Memorization

×10

2

3

2

1

0

1

2

Memorization

×10

2

6

4

2

0

2

Memorization

×10

2

Pythia

70M

Mid

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Memorization

×10

2

1.5

1.0

0.5

0.0

0.5

Memorization

×10

2

6

5

4

3

2

1

0

1

2

Memorization

×10

2

Pythia

70M

Late

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0

1

2

3

4

Memorization

×10

2

2

1

0

1

2

3

4

Memorization

×10

2

4

2

0

2

4

Memorization

×10

2

Pythia

410M

Mid

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

62

64

66

68

70

72

74

76

78

80

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0

1

2

3

4

5

6

7

Memorization

×10

2

2

0

2

4

6

Memorization

×10

2

4

2

0

2

4

6

Memorization

×10

2

Pythia

1B

Mid

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

125

128

131

134

137

140

143

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

0.5

0.0

0.5

1.0

1.5

2.0

2.5

Memorization

×10

2

2

1

0

1

2

Memorization

×10

2

6

4

2

0

2

Memorization

×10

2

Pythia

1B

Late

G.2 Fraction of Samples Above 50th Percentile

GPT-2

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 1.0

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

1

3

5

7

9

11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

1.0

Frac. above 50th percentile

120M

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.3

0.4

0.5

0.6

0.7

0.8

Frac. above 50th percentile

0.3

0.4

0.5

0.6

0.7

0.8

Frac. above 50th percentile

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Frac. above 50th percentile

Pythia

70M

Early

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.40

0.45

0.50

0.55

0.60

0.65

Frac. above 50th percentile

0.40

0.45

0.50

0.55

0.60

0.65

Frac. above 50th percentile

0.40

0.45

0.50

0.55

0.60

Frac. above 50th percentile

Pythia

70M

Mid

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

0.495

0.500

0.505

0.510

0.515

Frac. above 50th percentile

0.48

0.49

0.50

0.51

0.52

Frac. above 50th percentile

0.44

0.46

0.48

0.50

0.52

0.54

Frac. above 50th percentile

Pythia

70M

Late

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

Frac. above 50th percentile

Pythia

410M

Mid

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

Frac. above 50th percentile

0.2

0.4

0.6

0.8

1.0

Frac. above 50th percentile

Pythia

1B

Mid

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Frac. above 50th percentile

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Frac. above 50th percentile

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Frac. above 50th percentile

Pythia

1B

Late

G.3 Fraction of Samples Below 10th Percentile

1

3

5

7

9 11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 1.0

1

3

5

7

9 11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

1

3

5

7

9 11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

1

3

5

7

9 11 13 15 17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

10

2

10

1

Frac. below 10th percentile

GPT-2

120M

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

1

3

5

7

9

11

13

15

17

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

10

1

Frac. below 10th percentile

10

1

Frac. below 10th percentile

10

2

10

1

Frac. below 10th percentile

Pythia

70M

Early

63 65 67 69 71 73 75 77 79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63 65 67 69 71 73 75 77 79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63 65 67 69 71 73 75 77 79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

10

1

6×10 2

Frac. below 10th percentile

10

1

6×10 2

Frac. below 10th percentile

10

1

6×10 2

Frac. below 10th percentile

Pythia

70M

Mid

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

10

1

9.4 × 10

2

9.6 × 10

2

9.8 × 10

2

1.02 × 10

1

1.04 × 10

1

Frac. below 10th percentile

10

1

8.5 × 10

2

9×10 2

9.5 × 10

2

1.05 × 10

1

1.1 × 10

1

Frac. below 10th percentile

10

1

7×10 2

8×10 2

9×10 2

Frac. below 10th percentile

Pythia

70M

Late

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

Pythia

410M

Mid

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.1

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.01

63

65

67

69

71

73

75

77

79

1

3

5

7

9

11

13

15

17

Data segment

Sampling Rate 0.001

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

Pythia

1B

Mid

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.1

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.01

126

129

132

135

138

141

1

4

7

10

13

16

Data segment

Sampling Rate 0.001

10

4

10

3

10

2

10

1

Frac. below 10th percentile

10

4

10

3

10

2

10

1

Frac. below 10th percentile

10

3

10

2

10

1

Frac. below 10th percentile

Pythia

1B

Late

G.4 Memorization Delta Histograms

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Density

Checkpoint #1

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Density

Checkpoint #2

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #3

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

12

Density

Checkpoint #4

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

12

14

Density

Checkpoint #5

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

2.5

5.0

7.5

10.0

12.5

15.0

Density

Checkpoint #6

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Density

Checkpoint #7

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Density

Checkpoint #8

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #9

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #10

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #11

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #12

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #13

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

Density

Checkpoint #14

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

Density

Checkpoint #15

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #16

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #17

train set

valid. set

GPT-2

120M

3

2

1

0

1

2

3

Memorization Delta

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Density

Checkpoint #1

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #2

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

12

Density

Checkpoint #3

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Density

Checkpoint #4

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #5

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

Density

Checkpoint #6

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #7

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #8

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #9

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #10

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #11

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #12

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #13

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #14

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #15

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #16

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #17

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

Density

Checkpoint #18

train set

valid. set

Pythia

70M

Early

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #63

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

Density

Checkpoint #64

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

Density

Checkpoint #65

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #66

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

7

Density

Checkpoint #67

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

10

Density

Checkpoint #68

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #69

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #70

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #71

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #72

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

7

Density

Checkpoint #73

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #74

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #75

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #76

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #77

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #78

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

2

4

6

8

Density

Checkpoint #79

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #80

train set

valid. set

Pythia

70M

Mid

Pythia

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Density

Checkpoint #126

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

Density

Checkpoint #127

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #128

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Density

Checkpoint #129

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

Density

Checkpoint #130

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #131

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #132

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

6

Density

Checkpoint #133

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Density

Checkpoint #134

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

Density

Checkpoint #135

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #136

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #137

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #138

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

Density

Checkpoint #139

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #140

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Density

Checkpoint #141

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

1

2

3

4

5

Density

Checkpoint #142

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Density

Checkpoint #143

train set

valid. set

70M

Late

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #63

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #64

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #65

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #66

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #67

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #68

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #69

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #70

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #71

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #72

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #73

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #74

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #75

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #76

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #77

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #78

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #79

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #80

train set

valid. set

Pythia

410M

Mid

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #63

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #64

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #65

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #66

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #67

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #68

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #69

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

5

10

15

20

25

30

35

Density

Checkpoint #70

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #71

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #72

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #73

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #74

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #75

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #76

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #77

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #78

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #79

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

Density

Checkpoint #80

train set

valid. set

Pythia

1B

Mid

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #126

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #127

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #128

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #129

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #130

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #131

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #132

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #133

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #134

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #135

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #136

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #137

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #138

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #139

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

Density

Checkpoint #140

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

Density

Checkpoint #141

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

70

Density

Checkpoint #142

train set

valid. set

0.4

0.2

0.0

0.2

0.4

Memorization Delta

0

10

20

30

40

50

60

Density

Checkpoint #143

train set

valid. set

Pythia

1B

Late

H More attack plots

H.1 Interpolation Attack

We repeat the interpolation attack experiment in the main body of the paper, with the 1B-parameter Pythia model and observe from Figure 36 that indeed the interpolated checkpoints fail our memorization tests.

65

70

75

80

2.165

2.170

2.175

2.180

2.185

2.190

Validation Loss

Original Run

Interpolated Run

0

2

4

6

8

10

12

14

16

18

1

3

5

7

9

11

13

15

17

Data segment

Memorization

2

0

2

4

6

×10

2

̸

H.2 Data Subtraction Attack Tests

In the following subsections, we plot the subtraction-upper-bound heuristic λ (Π i , p, W i ) with varying values of p , for different subtraction rates. We observe that for big enough models λ is a tight upper-bound when no subtraction has happened. For Pythia with 70M parameters, our smallest model, λ does not provide a tight upper-bound. However, for GPT-2 with 124M parameters, Pythia with 410M parameters, and Pythia with 1B parameters, λ provides a tight upper-bound.

For the 1B-parameter Pythia model, we further plot the upper-bound heuristic for varying values of the checkpoint interval (number of training steps between each checkpoint). From Figure 41, we observe that even though λ increases as the interval increases, it is still a good upper-bound ( ∼ 0.05 for a checkpoint interval of 5000 steps) for p = 0 . 1 and p = 0 . 2 . This means that we can save checkpoints less frequently and still use the heuristic to detect data subtraction.

H.2.1 GPT-2

5

10

15

0.00

0.25

0.50

0.75

1.00

1.25

Subtraction Lower-Bound

Full Data

5

10

15

0.00

0.25

0.50

0.75

1.00

1.25

5% Subtraction

5

10

15

0.00

0.25

0.50

0.75

1.00

1.25

50% Subtraction

5

10

15

0.00

0.25

0.50

0.75

1.00

1.25

95% Subtraction

p

=0.01

p

=0.1

p

=0.2

p

=0.3

p

=0.4

p

=0.5

H.2.2 Pythia (70M)

65

70

75

80

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Subtraction Lower-Bound

Full Data

65

70

75

80

0.00

0.25

0.50

0.75

1.00

1.25

1.50

5% Subtraction

65

70

75

80

0.00

0.25

0.50

0.75

1.00

1.25

1.50

50% Subtraction

p

=0.01

p

=0.1

p

=0.2

p

=0.3

p

=0.4

p

=0.5

H.2.3 Pythia (410M)

65

70

75

80

0.0

0.2

0.4

0.6

0.8

Subtraction Lower-Bound

Full Data

65

70

75

80

0.0

0.2

0.4

0.6

0.8

5% Subtraction

65

70

75

80

0.0

0.2

0.4

0.6

0.8

50% Subtraction

p

=0.01

p

=0.1

p

=0.2

p

=0.3

p

=0.4

p

=0.5

H.2.4 Pythia (1B)

65

70

75

80

0.0

0.2

0.4

0.6

0.8

1.0

Subtraction Lower-Bound

Full Data

65

70

75

80

0.0

0.2

0.4

0.6

0.8

1.0

5% Subtraction

65

70

75

80

0.0

0.2

0.4

0.6

0.8

1.0

50% Subtraction

p

=0.01

p

=0.1

p

=0.2

p

=0.3

p

=0.4

p

=0.5

65

70

75

80

0.00

0.05

0.10

0.15

0.20

0.25

Subtraction Lower-Bound

Checkpoint Interval = 1000

64

66

68

70

0.00

0.05

0.10

0.15

0.20

0.25

Checkpoint Interval = 2000

63

64

65

66

67

0.00

0.05

0.10

0.15

0.20

0.25

Checkpoint Interval = 3000

63.0

63.5

64.0

64.5

65.0

65.5

66.0

0.00

0.05

0.10

0.15

0.20

0.25

Checkpoint Interval = 4000

63.0

63.5

64.0

64.5

65.0

0.00

0.05

0.10

0.15

0.20

0.25

Checkpoint Interval = 5000

p

=0.01

p

=0.1

p

=0.2

p

=0.3

p

=0.4

p

=0.5

I Broader Impacts

We intend this work to be a step towards meaningful and transparent public oversight of large AI systems, especially those with capabilities whose irresponsible use could significantly harm the public. Our protocol is a sketch of a technical framework for a system by which AI developers can prove properties of their training data, and may thereby enable the effective enforcement of a broader set of policies than those solely relying on querying models 'black-box'. While enabling many possible positive rules, this could also be misused by coercive states to detect and enforce harmful restrictions on beneficial AI development. However, in most cases, such authoritarian states would already have a means for policing domestic AI developers' behavior, and verification tools demanding so much cooperation from the Prover are unlikely to meaningfully increase existing surveillance powers. Another issue is that requirements for complying with monitoring and enforcement tend to favor large companies, for whom the cost of compliance can more easily be amortized. This motivates efforts to keep verification schemes simple, flexible and cheap.

We hope that this protocol can also be useful for verifying agreements between untrusting countries. The protocol itself does not provide a means for identifying that an AI model was developed in the first place unless it is disclosed. In this sense, it more closely parallels a process for an AI-developing country to allow its counterpart to retroactively inspecting a developed system (paralleling the New START treaties' inspections of nuclear launchers), rather than to proactively detect when a new system is deveoped (paralleling the IAEA's monitoring of the process of uranium enrichment).

Because our protocol supports multiple independent auditors reviewing the same transcripts, we hope that these tools will support the development of trust between competing companies and countries. Ultimately we hope such protocols will support the development of a larger governance ecosystem representing many parties.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a protocol for verifying that a model trainer submits data and learned weights, and a verifier checks whether the weights are correctly learned from the submitted data.\nSuch a protocol is useful for trustworthy AI. The paper defines the problem of Proof-of-Trainind-Data (PoTD). It is inspired by the previous work on \"Proof-of-Learning,\" but the setting of PoTD requires more. The paper gives a formal definition of PoTD protocol and argues that the protocol needs to satisfy some conditions to achieve the guarantee in a practical setting.\n\nSTRENGTHS:\nThe motivation of the paper is interesting. The proposed PoTD protocol seems reasonable.\n\nWEAKNESSES:\n1. The paper does not explain the motivation why we need the PoTD protocol and its impact. The paper says that some attacks can be treated by existing Proof-of-Learning methods. It seems trivial since  PoTD poses stronger requirements, as written in the paper. The paper should discuss the problems of existing PoL methods and how the proposed protocol can solve them.\n\n2. The only attack that the existing method cannot deal with but the proposed method can is a data subtraction attack. How important to deal with such an attack is not explained enough.\nMoreover, experimental results show how the proposed method performs but do not compare with baseline methods. Therefore, experiments are not enough to\nshow the effectiveness of the proposed methods. \n\n3. The presentation of the paper is hard to follow. The paper seems to consist of fragments of texts relationships between them are unclear. \nFor example, There is a formal definition of PoTD protocol in section 2, but the definition is never mentioned in the subsequent sections. Therefore, it is hard to judge whether the definition is reasonable or not. \nThe memorization heuristic introduced in Section 3.2 seems overly complex. If we use $-L(d, W)$ instead of M(d, W) in (3), then the values of PBQ and FBQ would not change since\nthe first term of (2) does not depend on data d and has no effect when evaluating $\\Delta_M(d^\\prime, W) >  \\Delta_M(d, W)$.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nThe paper mentions to limitations in the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Proof of Training Data (PoTD), a variant of Proof-of-Learning (PoL) protocols that focuses on training set attacks, rather than the training algorithm itself. A valid PoTD protocol should be able to, at least in theory, spot when a machine learning model has been trained on a different training set than the one declared by the learner. As in PoL, the learner is required to provide a full transcript of the training process, including training data, code and intermediate checkpoints. Unfortunately, the task of verifying a training transcript is as computationally intensive as re-training the model from scratch. For this reason, the authors propose several heuristic strategies for PoTD, which rely on the fact that stochastic gradient ascent tends to first memorize and then forget the data it observes in each batch.\n\nSTRENGTHS:\nFrom the methodology standpoint, I truly appreciate the memorization heuristics proposed in the papers. In particular, they seems to be able to defend against a large number of different threat models.\n\nWEAKNESSES:\nThe PoTD protocol proposed by the authors have considerable overlap with the existing PoL proposals. I am inclined to see it as a variant of these existing efforts, rather than a novel, independent idea.\n\nFurthermore, the defenses proposed in the paper are heuristics and have been tested on large language models only. The authors mention that their techniques may work differently on other neural architectures, learning tasks and training procedures. Therefore, I believe more experimentation is needed to confirm their usefulness.\n\nAdditionally, the PoTD protocol requires the learner to disclose their complete learning process. Thus, there is no way to protect the intellectual property of the learner. The authors are honest about this limitation, and claim it will be addressed as future work. However, I feel this makes the paper weaker.\n\nThe data addition attack presented in lines 256-267 seems the most interesting scenario to me. It is unfortunate that the heuristic technique proposed by the authors cannot defend against it.\n\nThe writing style of the paper could be improved. I report here some specific examples.\n\nLine 6, \"and flag if the model specific harmful or beneficial data sources\" is not a syntactically-correct English sentence. \n\nLine 30, the authors start discussing how to solve proof-of-training-data\nbefore giving a precise enough definition that the reader can follow.\n\nLine 87, c2 is missing from the definition of V (compare with Line 82). Also, why is the probability taken over c1, when c1 does not appear in any of the terms?\n\nLine 172-173, please number all equations.\n\nSection 3.3 is very dense and many important details are left for future work.\n\nFootnotes 4-9 occupy almost a quarter of the page and contain important information. I would prefer having them merged with the main text.\n\nLine 225, \"on trained\" should be \"trained on\".\n\nSection 5 introduces new concepts, new attacks, new defenses, new notation. Since this happens so late in the paper, it ends up being a bit overwhelming. Why not explicitly organise the whole paper by threat model?\n\nQUESTIONS:\nWould it be possible to bypass the memorization heuristic by changing both the model and the data? Theoretical work has shown that only a small portion of a large neural network is important for achieving good predictive performance (lottery tickets). Once a learner has trained a good model on dataset D*, it should be possible to add (a large number of) redundant neurons and retrain those on D. Would this be a valid attack on PoTD?\n\nThe Equation between lines 172 and 173 is not obvious at first sight, and should be clarified. Also, please number all equation in the paper.\n\nLIMITATIONS:\nI commend the authors for beign upfront about the limitations of their work. All my concerns have been listed above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper presents a novel protocol called Proof-of-Training-Data, which a third party auditor can verify the data used to train a model. Here, the auditor will require training data, training code, and intermediate checkpoints. Experiments on two language models have demonstrated that known attacks from the Proof-of-Learning literature can be caught by this new protocol.

STRENGTHS:
This paper attempts to tackle an important security problem on trained neural network models. The proposed heuristics (i.e., memorization-based tests) is appealing and can efficiently catch spoofed checkpoints using a small amount of data. The paper is adequately structured and solid experiments have been carried out to empirically justify the effectiveness of the proposed protocol.

WEAKNESSES:
The concept of Proof-of-Learning has been well studied. Although the authors have discussed various Proof-of-Learning literature in the related work and the experiments section, it is still not immediately clear to me why we need this brand-new protocol (Proof-of-Training-Data). If I understand correctly, the authors are attempting to solve an even harder problem where the adversaries can have more computing power.

QUESTIONS:
- Line 77, could there be a comparison between the formal formulation of Proof-of-Learning and Proof-of-Training-Data?
- Section 4, as most of the Proof-of-Learning experiments have been done on image datasets like CIFAR, I am curious if PoTD could perform similarly well on those image datasets.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes techniques and tools that can be used for verifying the \"provenance\" of large neural models, to evaluate their risks. These techniques and tools are part of \"protocols\" used by a model trainer to convince a \"verifier\" that the training data was used to produce the model parameters. The authors show experimentally that their prescribed procedures can catch a variety of known attacks from the \"proof-of-learning\" literature.\n\nSTRENGTHS:\n* The paper addresses an increasingly important problem, as large neural models are becoming very popular, and advances practical techniques that can be used by regulators to check the provenance of large models.\n\n* The authors present convincing evaluation using GPT-2, demonstrating that the proposed procedures are effective in catching a variety of attacks, such as glue-ing and interpolation as well as data addition and subtraction.\n\nWEAKNESSES:\n* I find that the use of \"proofs\" in the title and throughout the paper is misleading as the authors do not present techniques that amount to an actual proof.\n\n* I think it is great that the techniques presented by the authors are practical and effective wrt several attacks but I wonder if this is enough for regulators. I mean they would possibly need stronger guarantees for such techniques. \n\n* It is unclear to me how the verification strategies presented in section 3 relate to definition 1. The authors should work on adding a theorem that clearly states that their strategies achieve the desired properties, i.e., the verifier accepts/rejects true winesses/spoofs with the desired probabilities.\n\n* The technical contribution beyond \"proof-of-learning\" is unclear.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nIn conclusion this is very interesting work but may be too preliminary for publication.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}