Tools for Verifying Neural Models' Training Data
Authors: Dami Choi
Yonadav G Shavit
David Duvenaud
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Scale Learning
ML Security
AI Governance
TL;DR: We show ways to tell if people trained large models on the data they said they did.

Abstract: It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a "Proof-of-Training-Data": any protocol that allows a model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of-Training-Data that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to a random seed used in training, and a method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point was included in training. We show experimentally that our verification procedures can catch a wide variety of attacks, including all known attacks from the Proof-of-Learning literature.

Tools for Verifying Neural Models' Training Data
Dami Choi"
Yonadav Shavit"
David Duvenaud
U. Toronto & Vector Institute
Harvard University
U. Toronto & Vector Institute
choidami@cs. toronto.commmmm edu
yonadav@g.harvard.com harvard.
duvenaud@cs toronto................ edu
Abstract
It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a "Proof-of- "Proof-of-Trainnng Traininnn : any protocol that allows model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of- -ooo---raiinn---- -ata that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to random seed used in training, and method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point We show experimentally that our verification procedures can catch wide variety of attacks, including all known attacks from the Proof-of-Learning literature.
included in training.
Introduction
How verify the capabilities of large machine learning models? Today, such claims based on trust and reputation: customers and regulators believe that well-known companies building AI models wouldn't tt lie about the training data used in their models. However, the ability build new AI models proliferates, users need to trust an ever-larger array of model providers at their word, and regulators may increasingly face malicious AI developers who may lie appear compliant with standards and regulations. Worse, countries developing militarily-significan AI systems may not each others' claims about these systems' capabilities, making hard coordinate on limits.
AI developers can enable greater trust by having a third party verify the developer's claims about their system, much as the iOS App Store checks apps for malicious code. Current black-box approaches to model auditing allow some probing of capabilities [Cen23], but these audits' utility is limited and a model's 's capabilities can be hidden [GTB22, GKVZ22). An auditor more effectively target their examination they also know the model's training data, including the total quantity, inclusion of data likely enable specific harmful capabilities (such texts cyber-exploit generation), and data (such as instruction-tunm tuning [OWJ+22]] 22]). However, if auditors rely
inclusion of safety-enhancee on AI developers to self-report the data, developers could falsify their reports. This uncertainty limits the trust such audits can create.
In this work, define the problem of Proof-of-Trainiee Taat--rr-nnnnnaatt (PoTD): protocol by which third- party auditor (the "Verifier") can verify which data was used by the developer (the "Prover") to train a model. Our verification procedures assume that the Verifier can be given access to sensitive information and IP (e.g., training data, model weights) and is trusted to keep it secure; we leave the additional challenge of simultaneously preserving the confidentiality of the training data and model weights to future work. In principle, one could solve PoTD by cryptographically attesting to the
"Equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

results of training on dataset using delegated computation [CKV10]. However, in practice such delegation methods are impractically slow, forcing us to turn heuristic verification approaches.
Inspired by the related literature on "Proof-of-Leerning trainers disclose training transcript the Verifier, including training data, training code, and intermediate checkpoints. In Section 4, we provide several verification strategies for Verifier to confirm a training transcript's authenticity, including new methods that address all published attacks in the Proof-of-Learning experiments on two language models (Section 6). Our methods can be run cheaply, adding as little as 1.3% of the original training run's 's compute. Further, we require no change to the training pipeline other than fixing the data ordering and initialization seeds, and storing the training process seeds for reproducibility. Still, like PoL, they sometimes require re-running a small fraction of training steps to produce strong guarantees.
(PoL))ICCC*21), we propose that model-
literature. We demonstrate the practical effectiveness of our defenses via
The verification strategies we describe are not provably robust, but are intended as an opening proposal which we hope motivates further work in the ML security community investigate new attacks and defenses that eventually build public confidence in the training data used to build advanced machine learning models.
2 Related Work
We build on [Sha23], which sketches a larger framework for verifying rules on large-scale ML training. It defines, but does not solve, the to Soof-oof-Training Data that additionally requires verifying hyperparameters.
TaanTTaa---Taannneee problem, a similar problem
Proof-of-Learning. [JYCC+21] 21] introduce the problem of Verifier checks Prover's ownership/copyright claim over a set of model weights by requiring the Prover to prove that they did at least as much computational work as was required have trained the original model. The Prover's proof consists of reporting a sequence of valid weight checkpoints that led to the final weights. The Verifier then re-executes training between a few checkpoints and checks that the results are similar to confirm (or reject) the report's 's correctness. Unlike PoL. Proof-of- Training- Data requires proving what exact data was used produce model. This is strictly harder: any valid PoTD protocol can serve as solution to PoL., as compute cost can be derived from the size of the dataset. Furthermore, PoL. only requires robustness to adversaries that use less computation than the original training run, whereas PoTD targets all computationalllyfessiiii
(PoL), in which
adversaries.
Several works have successfuly attacked the original PoL scheme; our proposed defenses defeat all published attacks. !ZLD+22 22] and [KRCC22] provide methods for forging fake data that will (during retraining) interpolate between chosen checkpoints, thus allowing the production of fake transcripts. (See Section 4.3 for our mitigation.) [FJT* 22] exploit the fact that, due to the prohibitive cost of retraining, the Verifier can only retrain tiny fraction of checkpoints. By falsifying only modest fraction of checkpoints, a Prover can prevent their fake transcript being caught with high probability. (See Section 4.2 for our mitigation, which introduces a method for verifying checkpoints that much cheaper than retraining, and can thus be applied to all checkpoints.)
Memorization during training. [ZIL [2] introduce the notion of counterfactual memorization (the average difference in model performance with and without including a specific point in training) that most similar our own, and use it investigate different training points' effects on final model performance. [FZ20] examine which datapoints are most strongly memorized during training by using influence functions, but they focus on the degree of memorization only at the end of training. [BPS* 23] show that per-datapoint memorization of text (as measured by top-1 recall) can be some- what reliably predicted based on the degree of memorization earlier in training. [KGG 22] analyze pointwise loss trajectories throughout training, but do not focus specifically on the phenomenon of overfitting to points in the training set.
3 Formal Problem Definition
In the Proof-of- -raioff-rainnnn-aat problem, a Prover trains an ML model and wants to prove to Verifier that the resulting target model weights W are the result of training on data D" If malicious Prover used training data that is against the Verifier's 's rules (e.g.. terms of service, regulatory rules) then that

---

Prover would prefer to hide D' from the Verifier. To appear compliant, the Prover will instead lie and claim to the Verifier that they have used some alternative dataset D + D However, the Prover will only risk this lie if they believe that with high probability they will not get caught (making them a "covert adversary" [AL07]). The goal of a
Training- Daann protocol is to provide a series of Verifier tests that the Prover would pass with high probability if and only if they truthfully reported the true dataset that was used to yield the model W
Let D E X be an ordered training dataset. Let M contain all the hyperparameters needed to reproduce the training process, including the choice of model, optimizer, loss function, random seeds, and possibly details of the software/hardwarr configuration to maximize reproducibility.
Definition 1. A valid Proof-of cooff-of-Taiininn Training- Dattinng--aaaaaaaa protocol consists of Prover protocol P. Verifier protocol V. and witnessing template J that achieves the following. Given a dataset D and hyperpa- rameters M an honest Prover uses P to execute a training run and get (W* = P(D", M ..), where W E g is a final weight vector, J E J is witness to the computation, and 0₁ ~ Ci is an irreducible source of noise. The Verifier must accept this true witness and resulting set of model weights with high probability: (a.p=- "VD"", .M,,"" W2,02) = ò1 where 51 < 1/2 and C2 is the randomness used by the Verifier.
probabilistic adversaries A which produce spoofs M,J A(D" ,M,," W",, ,ca) where D * D and Cs C3 is the random- ness used by the adversary, the Verifier must reject all such spoofs with high probability: "aa=-i,ea--Caa---Caa (VD,,, M, W (2) 0) 2 1 82 where 82 & 1/22.
Conversely, A computationally-faatte
Following the literature on the related Proof-of-LLarnin problem [JYCC+21], we use as a witness the series of m model weight checkpoints J W (Wo Wi,..., W Wm-1, W*). Model weight checkpoints are already routinely saved throughout large training runs; we assume checkpoint is saved after training on each & = n/ n/mm-atapoint segment. During verification, the Prover provides the Verifier with the training transcript T = {D, M, W). which the Verifier will then test to check its truthfulness.
In practice, we cannot yet pursue provable robustness to all probabilistic adversaries A, due to the nascent state of the theoretical foundations of deep learning [FJT*22] 22]. Instead, as done the Pol. literature, we approximate robustness to A by proposing range of adversaries and then showing a protocol that defeats them. In particular, we will consider two major types of attacks (and their combinations):
Non-Uniqueness Attack: The Prover forges a different D * D and M (and corresponding checkpoints W) that would, if training was reexecuted, also lead to W*, In general, is easy to produce such transcripts (e.g.. initialize training at Wo = W* and take no steps). In Section 4.3 we propose constraints on P that block such attacks.
Data Subtruction Attack: The Prover reports training on D, but secretly only trains on a subset D* Ç D.
Data Addition Attack: The Prover reports training on D, but secretly trains on additional set of data D 2 D.
Checkpoint Glue-ing Attack: The Prover reports pair of successive checkpoints (W,, W++1) C W. but W++1 was not produced from W; by a training procedure at all.
We address the latter three attacks in Section 6.
As a brute-force solution to Proof-of- tooofoffTraining-Daie the Verifier could simply re-execute the complete training process defined by T, and check that the result matches W*. However, beyond technical complications", doing so is far too computationally expensive to be done often; a government Verifier would need to be spending as much on compute for audits as every AI developer combined. Therefore any verification protocol V must also be efficient, costing much less than the original training run. Inevitably, such efficiency makes it near certain that the Verifier will fail catch spoofs D A D* if D only differs in a few data points; in practice, we prioritize catching spoofs which deviate on a
"This would also fail in practice because of irreducible hardware-lewe noise which means that no two training runs return exactly the same final weight vector [JYCC*21], 21]. transcript could still be examined piecewise, as done in [JYCC+21 21]: for more, Section 4.1.
3

---

substantial fraction of points in D Though we do not restrict to particular definition of dataset deviations, we list several possibilities relevant for different Verifier objectives in Appendix D.
4 Verification Strategies
We provide several complementary tools for detecting whether transcript T is spoofed. Combined, these methods address many different types of attacks, including all current attacks from the PoL. literature [ZLD + 22, FJT+ 22].
4.1 Existing Tools from Proof-of-Learning
Our protocol will include several existing spoof- spoof-detection tools from the Proof-of-Leartin literature (JCC+*2]], such as looking for outliers in the trajectory of validation loss throughout training, and plotting the segment-wise weight-change |[W W;-1lla ll between the checkpoints W. The most important of these existing tools is the segment-wise retraining protocol of [FJT-22]. Let R( IT, c; M) be the model training operator that takes weight checkpoint W,,,, updates it with series of gradient steps based on training data sequence IT (describing the order in which the Prover claims data points were used in training between checkpoints W;-1 and Wa, which may be different from the order of the dataset D*), hyperparameters M. and hardware-noise- randomness e ~ C, and then outputs the resulting weight checkpoint W,. Transcript segment is (c, 8) reproduciee if for the pair of checkpoints (W,11, W,) in W, the reproduction enror (normalized by the overall segment displacement) is small:
I
Walla IIW. wI
Pr C W.
<e 11 where W; = R(W (I)
The values and trade off false-positive vs. false-negative rates; see [JYCC=21, 22] for dis- cussion. The Verifier can use this retraining procedure as a ground-trui truth for verifying the faithfulness of a suspicious training segment. However, this test is computationally inteessivve and can thus only be done for a small subset of training segments. Our other verification strategies described in Sections 4.2 and 4.3 will be efficient enough to be executable on every training segment.
4.2 CemoriiaationBassee Tests
The simplest way for Prover to construct a spoofed transcript ending in W* is to simply make up checkpoints rather than training on D*, and hope that the Verifier lacks the budget to retrain a sufficient number of checkpoints to catch these spoofed checkpoints. To address this, a heuristic for catching spoofed checkpoints using a small amount of data, based on what is to the best of our knowledge previously-undee undocumentedd phenomenon about local training data memorization.
demonstrate
Machine learning methods notoriously overfit to their training data D, relative to their validation data D. We can quantify the degree of overfitting to single data point d on a loss metric L X x R W| + IR relative to a validation set D, via a simple memorization heuristic M:
M(d, W) = E. [C(d', W)] - L(d, W)).
(2)
Recall that 11, is the sequence of data points corresponding the ith segment of the training run. One would expect that in checkpoints before data segment i, for data points d E II, memorization M(d, Wyss) would in expectation be similar the validation-set memorization; after data-segment i, one would expect to see higher degrees of overfitting and therefore M(d. W,zs) would be substan- tially higher. We find evidence for this effect in experiments on GPT-2-Smail [RWC+19 19] and the Pythia suite [BSA + 23]). As shown in Figures and 2, when Prover reports the true training data, on average the greatest memorization occurs where IT, and W, match. We corroborate this finding with additional experiments on range of models in Appendix G. The finding is even clearer if we look at jumps in memorization level, which we call the Memorization Delta AM:
(m De, L) M(d. M(d, W,-1).
To test whether each reported checkpoint W. resulted from training on at least some of the segment training data 11,, Verifier can compute memorization plot like the one shown in Figure 1. Such

---

L065
Mamor salion
Memerization Delta
Hertor PenorOOaIOO
DVth Seit
30 32
12 36
32
37
-10
30
Chacspan:
Figure 1: Plots from a GPT-2 experiment, demonstrating the local memorization effect. The maximum score for each data segment (row) is marked with red box. The largest average memorization for data sequence II, occurs at the of the average loss L; memorization M; and memorization-dele A along with the average memorization over time for each segment II; recentered such that W; is at 0.
checkpoint W, From left to right: plots
1550
t
22
4000
-23
Checkpeint Num.
Checapontturn NUm
Checkssint Num.
Checspoin Nm
Figure 2: Plots of the memorization M other types of training runs, similar to Figure 1. For efficiency, plots of the Pythia models use only 10% of training data, and look only at a window of checkpoints around W,- From left right: M for checkpoints near the middle of the Pythia (70M) training run shows the same pattern as GPT-2; checkpoints near the middle of the Pythia (IB) training show that the phenomenon gets clearer as the model size increases; M for GPT-2 run with two epochs over the same data (with random data order each epoch; the first epoch ends at checkpoint 9) demonstrate that the effect present over multiple epochs; M for a GPT-2 run using a random data order other than II shows that the effect is tied to the training data sequence itself.
plots can be computed more efficiently by sampling only small fraction a of the training data II, and by plotting only a few checkpoints Wi-s:.... W for each segment IT,-
We can further harness this memorization phenomenon to test whether on segment i, rather than training the full claimed data sequence πI, and yielding W,, the Prover secretly skipped training on at least a k- x-fraction of the points in II, and yielded W'. Consider the odds that, for d ~ Il, A M (d, W:) happens fall in the bottom p-probability quantile of the validation D AM values on Wi:
PBQ W.) = II (E I(Am(d) W:) > Am W.)) S P) FBQ p. W.) Eą-H [BQ(dd, Wo),
where II is the indicator function, PBQ stands for "Point Below Quantile", and FBQ stands for "Fraction Below Quantile". We can see Figure 3 that, as expected, when the points II; are all included in training, FBQ p, W.) is small compared to FBQ W:)). If many points were secretly excluded, as in W', we should expect that FBQ(II, p, W/) should be higher and closer FBQ(D., p. W?), where D,, the validation
If the Prover secretly excluded a greater than i- e-fraction of data points in II, thus yielding W', then we should expect that:
FBQ(I;; W!) = (1 K) F)B P. W.) + xFBQ(D,, D.., W!) 2 FBQ(D W()
(6)
(see derivation in Appendix B). Rearranging terms, we get
A(TI, P. WT) = FBQ p, W!)
2 k.
FBQ(D, p. W?)
X(11 W?) can be estimated using small fraction of training and validation datapoints, and can serve an upper-bound estimate
the fraction of II, secretly excluded from training WT.3 In
'The percentile- threelhlld is left unspecified, but should be kept & 0.5. The test can be strengthened by varying the chosen fraction p and rerunning the analysis confirm its insensitivity.
5

---

Prec. of points above 50th percentlee
Frac. of ponta be ow 12th parce e0
Checkpoint
20
Checkpo it #9
1.25
1.00
25
0.50
13
11
Apn 0.25
37
17
-5.2 0.0 5.2 2.4
Momorfaaionn
Figure 3: Exploring the pointwise memorization effect on GPT-2. From left to right: For each W; and HI, we plot the fraction of points with AM above the median 1 - FBQ(II, 0.5, W.), and see that in the diagonal segments, most individual points are above the median. This shows that memorization occurs pointwise, suggesting that it can be detected via sparse random sampling. The highest segment in each row is surrounded by a red box; Plotting the fraction of samples below the 10%ile, we see the fraction is uniquely low on diagonal tiles (II, Wj=i), predicted; two histograms comparing AM for diagonal vs. nondiagonal weight checkpoints across all data segments, shows how the distributions may may not overlap. Even at Checkpoint 1, the leftmost plot shows that II has (marginally) larger fraction of points above the median than any other data segment.
Section 6 we show that this heuristic can detect even small data subtractions in practice, and in Appendix H.2 we show the test's effectiveness across a range of percentiles p and segment-lengths
We also observe that .M gradually decreases over time from an initial peak immediately after the point's training segment. This echoes the many findings "forgetting" in deep learning (TSC+ 18). We show in Section how this can be used to catch gluing attacks.
4.3 Fixing the Initialization and Data Order
As mentioned in Section 3, Proof-of-Trainne Training-DDaa protocol needs to defend against non-uniquness attacks, by making it difficult for malicious Prover to produce second transcript with * D that, if training was legitimately executed, would also end in W There are two well-known types of attacks the Prover might use to efficiently produce such spoofs:
Initialization attacks: An attacker can choose a "random" initialization that places Wo in a convenient position, such as close to the target W*. Even if the Verifier uses statistical checks to confirm that the initialization appears random, these are sufficiently loose that an adversary can still exploit the choice of initialization [ZLD+222].
Synthetic data/data reordering attacks: Given the current weight vector W,, an attacker can synthesize a batch of training datapoints such that the resulting gradient update moves in a direction of the attacker's choosing, such towards W This could be done through the addition of adversarial noise to existing data points [TJSP22], or by carefully reordering existing data points "reordering attack" [SSK+21) 21].
generating a new dataset
We propose methods for preventing both of these attacks by forcing the Prover to use a certified- random weight initialization, and certified-randon data ordering. The randomized data ordering guarantees that the adversarial Prover cannot construct synthetic datapoints that induce a particular gradient, because it does not know the corresponding weights W at the time of choosing the datapoints D. Given a fixed data ordering, we discuss in Appendix E why it may be super-polynomially hard to find a rertified-randoom weight initialization that, when fully trained, results in particular W
The Verifier can produce this guaranteed to use particular random seed s, constructed as a function of the dataset D itself. This produces the initialization Wn = (s) E X and data ordering S = Gp(s) using a publicly known pseudorandom generators G, and Gy.* 4 The Prover can also construct a verifiable validation subset D, by holding
initialization and data order by requiring the Prover
'G, is sespporrrphicallyseeem pseudorandom d-length vector generator, with postprocessing defined in the hyperparameters M, and G, is Aiblicly-agree pseudorandom re-lenghh permutation generator. G, can be modified repeat data multiple times train for multiple epochs, according
randomized curriculum. "In practice, the statistical test to verify that the certified ordering was used will only be able to distinguish whether each data point de D was trained in the assigned segment S, or not. Therefore, for this protocol to apply checkpoint must be saved at least twice per epoch, A n/2.

---

out the last ny dlat-ppoin in the permutation S from training. The Prover constructs as follows. Assume that the dataset D has some initial ordering. Let H be publicly-knon -nnliclly-kwwwnnnn cryptographic hash function. We model H as random oracle, so that when composed with G, or Gp. the result is polynomial-time indistinguishable from random oracle. This means that if a Prover wants to find two different seeds $1, $2 that result in similar initializations Wo;1, Wa,2 or two similar permutations S1. S2. they can find these by no more efficient method than andesinnnnd.. For large d and n., finding two nontrivially-reateed random generations takes exponential time. We construct the dataset-dependent random seed as
D, Brand) = H (H(d)) o ((22) ..... o H(da) Brand).
(8)
where {di,.... do} D. is the concatenation operator, and Srand is a Prover-chosen 32-bit random number to allow the Prover to run multiple experiments with different seeds.? ? A Verifier given access D (or only even just the hashes of D) can later rederive the above seed and, using the pseudorandom generators, check that it produces the reported Wo and S.
The important element of this scheme is that given initial dataset D and resulting data order S, modifying even a single bit of single data point in D to yield a second D will result in a completely different data order S' that appears random relative to S. Thus, if we can statistically check that sequence of checkpoints W matches a data order 5 and dataset D) better than a random ordering, this implies that D is the only eeciiently-dicooveralll result in the checkpoints W and final weights W We provide this statistical test in Appendix C.
dataset that, when truthfully trained would
This same approach can be extended to the batch-online setting, where Prover gets sequence of datasets Di, Di,.... and trains on each before seeing the next. The Prover simply constructs a new seed s (D; Srand) for each dataset D and continues training using the resulting data ordering. This works so long as each D; is large enough for a particular data-ordering to not be frute-forceable.
4.4 Putting It All Together
In Appendix A sketch complete protocol for combining these defenses complementarily to detect all of the attacks discussed in Section 6. The overall computational cost for the Verifier is Ó(n) training data-point hashes, ((n)) model inferences for computing losses, and (([Q)n) (n) gradient computations for retraining transcript segments (where |Q] depends on hyperparameters that can be adjusted according the Verifier's compute budget). Importantly, the Verifier's cost grows worse than linearly with the cost of the original training run. If we run our tests using an a = 0.01 fraction of the points in each segment as done in our experiments below, the verification cost of computing our new tests in Sections 4.2 and 4.3 totals just 1.3% of the original cost of training, assuming inference 3x cheaper than training.
5 Experimental Setup
Our main experiments are run on GPT-2 [RWC 19] with 124M parameters and trained on the OpenWebText dataset [GCPT19]. We use a batch size of 491,520 tokens and train for 18,000 steps (~8.8B tokens), which is just under epoch of training, saving a checkpoint every 1000 steps. See Appendix F for additional details. The data addition attack experiments in Section 6 further use the Github component of the Pile dataset [GBB* 20] as a proxy for a Prover including additional data that is different from reported data. In addition to training our own models, we also evaluate Pythia checkpoints [BSA [AS)++33 +23] published by EleutherAl, as they publish the exact data order used to train their models. We chose the 70M, 410M, and IB-sized Pythia models trained on the Pile dataset with deduplication applied. All experiments were done using 4 NVIDIA A40 GPUs.
'Since random oracle model is known to be unachievable in practice, we leave the task of finding a more appropriate cryptographic primitive as an interesting direction for future work. 'To enable Prover only reveal the required subset of data the Verifier. it may be best construct using a Merkle hash tree.
'It is still possible to construct multiple data sets Dr, D2, and train on both, interleaving batches. This is not uniqueness attack, but data addition attack, and will be addressed in Section 6.
7

---

Empirical Attacks and Defenses
Below, we show that our methods address the last three attacks (Checkpoint Glue-ing, Data Sub- traction, and Data Addition). We omit the synthetic initialization and synthetic data attacks of 22J =LD=2 as we addressed those in Section 4.3. All plots are from experiments using 22] GPT-2; we include additional experiments in Appendix H. We do not claim that the attacks studied here are exhaustive, but provide them as starting point to motivate future work.
Checkpoint Glue-ing Attack A known attack against Proof-of-Leariigg which also applies to PoTD, is to "glue" two training runs W and W together and report a combined sequence of checkpoints W (W61 ,....y W.', W j.:::..- .............. Winaa) The resulting model W finat can be trained undisclosed data prior to segment with the Prover never reporting this data to the Verifier. As highlighted by [JYC the size of the glued segment [ Wr will generally appear outlier in weight-space. We demonstrate this phenomenon in Figure 4. Following [JYCC*21], a Verifier could then check such suspicious segments via retraining. We demonstrate second verification option using inference instead of training: the Verifier can check whether the checkpoint W.P has memorized not only the most recent data II, but also the preceding data segments 11,-1,11,-2 2., The absence of long-temn term memorization is visible in the memorization heatmap in Figure 4.
400
Memorization
x10
X10
300
Segmert
Mumorizaioon
250
200
11
150
100
50
1
3
12 13 15 27
4
8 Checkpoint Num.
6
10
12
14
16
18
5 20
15
Chockpaint Num. 60
Chockpaint Num.
Figure 4: Exploring how the defenses handle a simulated gluing attack, where the transcript switches from one GPT-2 training run to a second run after the 9th checkpoint. (We assume the Prover uses a certified data order (Section 4.3) the sections before and after gluing.) From left to right: The norm of the weight-changes jumps abruptly during the gluing, causing the Verifier to flag that checkpoint as suspicious.; the Verifier creates a memorization plot (shown here with 100% sampling rate for clarity), , and discovers the gluing by spotting that memorization past checkpoints cuts off abruptly the suspicious segment.; The same long-term memorization cutoff effect is visible plotting average M for each data segment across time.
To avoid the spike in weight-space shown in Figure 5 when jumping from W.A to W.B, the attacker can break up the large weight-space jump into smaller jumps by artificially constructing intermediate checkpoints aW + (1 W'WA for several values of a. However, these interpolated checkpoints fail our memorization tests, as they are artificial and not the result of actual training (Figure 5).
Data Subtraction Attack In data subtraction attack, a Prover claims the model has been trained on more points than it truly has. Detecting data subtraction attacks could enable Verifier detect overclaiming by model providers, including claiming to have included safety-related data when they secretly did not. Subtraction can also be used hide data addition attacks, as combining the two attacks would mean the segment was still trained on the correct number of datapoints, thus suppressing the weight-change-plos signature used to catch data addition (as in Figure 7). We demonstrate the effectiveness of an efficient memorization-basee based approach for detecting subtraction, described in Section 4.2. Leveraging the subtraction-upie wwwpetiiionuupp----eeeee upprro-ouddppd-dddddddoo bound test from Equation 7, we see in Figure 6 that the upper- ppunr-bound heuristic A(II, W;) is surprisingly tight, consistently differentiating between no-subtraction segments and even small subtraction attacks. Still, even if X(II,, p, W) > for some large > 0, this is only upper bound on the quantity of data subtraction, and does not prove that z-fraction of points were subtracted. The Verifier can instead use this test as an indicator to flag segments for retraining, which would confirm subtraction attack. (That retraining would
8

---

Memarization
4.0
Origina Intege ssad Fan
3.8
Data segment
3.6
Validalion
2.4
11-
3.2
13-
15-
3.0
17-
18
Chockpaint Num.
Chockpoint Num.
Figure 5: Simulating an interpolation attack by training GPT-2 model until the 5th checkpoint, and then
to a final checkpoint. On the left, we show that an attacker can carefully choose interpolation points to mask any irregularities in validation loss. (The green line perfectly overlaps with the blue line.) Nonetheless, on the right, we see a clear signature in the memorization plot, computed using only 1% of data: the typical memorization pattern along the diagonal does not exist for the interpolated checkpoints. For each row corresponding to a data segment II, a box marks the maximal-M checkpoint. The box red if the checkpoint is match W. and magenta if there is no match and the test fails Wjyi.
Sustraciion Test witsiis.pao p.1
Suitracion Test with p 0.2
Chectpont +1
Chedkocint eq
2.0
0.8.
6a% sittmetee
sutrrctionn
Subtreck Ja Lypen
0.2
13% rudiici
10s serrantinn
Denaky
0.6
1.0
0.4
0.5
20 Checieooie Rur.
-0.5
Checkpoint
Wemorizaton Detta
Nemorizatio
Figure 6: On the left, we simulate a data subtraction attack with different levels of subtraction (0%, 5%, 50%, or 95%) in a GPT-2 training run. The plots show the results of computing the subtraction- upper-bound heuristic A(II, p. W;) for each checkpoint, using just 1% of training data, across 20 random seeds, with dashed lines showing the true subtraction rate. À estimates the maximum level of data subtraction each segment. We see that provides a surprisingly tight upper bound for the honestly-trained segment, while providing no such upper bound for the larger subtraction attacks. To illustrate the logic behind this test, on the right, we show how a 50% subtraction attack can create a bimodal distribution of AM values. À captures the relative weight of the left mode.
result in a different weight vector can be inferred from the plot of the 50%-addition attack in Figure 7). Appendix H.2 explores the test's performance on the suite of Pythia models.
Data Addition Attack Addition attacks occur when a Prover in addition to training on the declared dataset D and data sequence II, trains on additional data D' without reporting it to the Verifier. This can be used to secretly enhance a model's 's capabilities, install backdoors, or train on restricted data. This attack cannot be detected using memorization analysis (Figure 7), because the Verifier does not know and cannot test points d' E D'. However, Figure 7 also shows that significant amount of data addition, or data addition from a different distribution ,causes a larger weight-displacement in that segment, which can be detected by looking at the segment's magnitude. If a Prover tries to hide this by deleting an cqually-sized subset of points from D, that can be detected as a subtraction attack. We leave exploration of other attacks and defenses, like changing learning rates, to future work.
This raises the problem of how to estimate the "expected" segment length, which may require retraining on a chosen subset of segments, and interpolating to estimate other segments' lengths. If, to hide the non-smoothness of adding extra data to single segment, the Prover adds data uniformly throughout a large fraction of the training run, then choosing even a small number of segments randomly should be sufficient to catch at least one offending segment with high probability. Unfortunately, these defenses would not detect an attacker that adds modest amount of data within a small number of segments, that cannot be detected by the regment-magninee or data subtraction defenses.

---

Memorization
X30
140
50% (OWT)
(OW/)
5% (tithul))
120
Bascfine (oWn)
Reproduction Error
0.8
Baseline (DWT)
Data segmenn
100
80
31-
60-
0.4
23-
40
15-
20
10 16
0.0
Chockpoint Num.
Checkpaint Num. 60
Checkpoint Num.
Figure 7: Simulating a data addition attack by picking single segment (Ist, 10th, or 18th), and adding 50% more data from the same distribution (Open WebText), or 5% more data from a different distribution (Github), or no data addition (truthful reporting). Results are shown with error bars across 4 random seeds. From left to right: the memorization test with 1% of samples does not spot any differences even with 50% data addition; Plotting weight-changes between checkpoints, a Verifier can see a suspicious spike at the attacked segments; The Verifier retrains the suspicious segments and checks the distance between the reported and re-executed checkpoint weights.
7 Discussion and Limitations
This work contributes to an emerging societal effort to develop practical and robust tools for account- ability in the large-scale development of AI models. The statistical tests we introduce are best taken as opening proposal. Future work could propose clever new attacks that break this protocol, or better yet, create new defenses that efficiently detect more, and subtler, attacks and enable trustworthy verification of ML models" training data.
Experimental Limitations This work provides suggestive evidence for the seccallmmmrrzationn phenomenon, but further study is needed across additional modalities, architectures, and training recipes in order to determine its broad applicability. Encouragingly, we find in Appendix G that hecalmmemriization gets even stronger as models get larger, though memorization appears weaker near the end of training as the learning rate shrinks. The paper's experiments only include language models, in part because they a current priority for audits. The memorization tests used may need to be adjusted for models trained with less data on many epochs, such as image models [YZS+22]
Attacks Our Protocol Does Not Catch There are several remaining directions for attacks. The attacks explored above can be composed in new ways, and it may be possible for compositions of attacks to undermine the defenses that would otherwise detect each attack individually. The method also does not address small-scale data additions, and thus cannot yet detect copyright violations or spot inserted backdoors [XWL + 1.]. It also cannot detect attacks based on small- -nor modifications to the weights, which could be used to insert backdoors [BISZ+22] Finally, attacks could be masked with cleverly chosen hyperparameters, such as by using a temporary tewpr-than-reporteed learning rate to shrink large changes in W. Exploring whether such attacks are feasible without degrading learning performance - and identifying defenses is an interesting direction for future work.
Applicability to Different Training Procedures We attempted to make our procedure agnostic as possible to the details of the training procedure, and believe it will be compatible with most training procedures for large models in use today. However, our protocol does not apply to online or reinforcement learning, or to schemes that require multiple models to be co-trained [GPAM 20], as the data is unknown in advance. This means the uniqueness defense cannot be applied (Section 4.3). Finding methods for defending against non- uniqueness attacks even in the online setting is a valuable direction for future work.
Maintaining Privacy and Confidentiality One significant challenge to using this protocol in practice is that, just like with PoL. [JYCC+21], it requires that the Prover disclose confidential information to the Verifier, including training data, model weights, and code. In principle, the Prover may only need to disclose hashes of the data and weights to the Verifier, with the matching full data and weights only ever supplied on the secure cluster during verification.
10

---

Acknowledgements
We thank Nicolas Papernot, Anvith Thudi, Jacob Austin, Cynthia Dwork, Suhas Vijaykumar, Rachel Cummings Shavit, Shafi Goldwasser, Hailey Schoelkopf, Keiran Paster, Ariel Procaccia, and Edouard Harris for helpful discussions. DC supported by NSERC CGS-D, and DC and YS are supported by Open Philanthropy AI Fellowships
References
[AHS22] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merg- ing models modulo permutation symmetries. arXiv preprint arXiv:2209.0488 2022.
[AL07] Yonatan Aumann and Yehuda Lindell. Security against covert adversaries: Efficient protocols for realistic adversaries. In Theory of Cryptography: 4th Theory of Cryptog- raphy Conference, TCC 2007, Amsterdam, The Netherlands, February 21-24, 2007. Proceedings 4, pages 137-156. Springer, 2007.
[BIZZ* 22] Mikel Bober- Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot. Architectural backdoors in neural networks. arXiv preprint arXiv:2206.07888 2022.
[3] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304. 158, 2023.
[BSA 23] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023.
[Cen23] Alignment Research Center. Update on ARC's recent eval efforts, March 2023.
[CKV10] Kai-Min Chung. Yael Kalai, and Salil Vadhan. Improved delegation of computation using fully homomorphic encryption. In Advances in Cryptology-CRYi 30th Annual Cryptology Conference, Santa Barbara, CA, USA, August 15-19, 2010. Proceedings 30, pages 483-501. Springer, 2010.
2010:
[FDRC20] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning. pages 3259-3269. PMLR, 2020.
[FJT+22] Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, and Nicolas Papernot. On the fundamental limits of formally (dis) proving robustness in proof-of-learning. arXiv preprint arXiv:2208. 03567, 2022.
[FZ20] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discov- ering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2881-288 Curran Associates, Inc., 2020.
[GBB [0] 20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace Hc, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
[GCPT19] Aaron Gokaslan, Vanya Cohen, Ellic Pavlick, and Stefanic Tellex. Openwebtext corpus, 2019.
[GKVZ22] Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable backdoors in machine learning models. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 931-942. IEFE, 2022.
[GPAM 20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-14) 2020.
[GTB22] Wei Guo, Benedetta Tondi, and Mauro Barni. An overview of backdoor attacks against deep neural networks and possible defences. IEEE Open Journal of Signal Processing, 2022.
11

---

[JYCC*21] Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Chore Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papemot. Proof-of-learnnng: Defi- nitions and practice. In 202/ IEEE Symposium Security and Privacy (SP), pages 1039-1056. IEEE, 2021.
[KGG+22] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning, arXiv preprint arXiv:2202.0999 2022.
[KRCC22] Zhifeng Kong, Amrita Roy Chowdhury, and Kamalika Chaudhuri. Forgeability and membership inference attacks. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security, pages 25-31, 2022.
[0W]+22] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[9WC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv preprint arXiv:1901.11199 2019.
[Sha23] Yonadav Shavit. What does it take to catch a Chinchilla? Verifying rules large-scale neural network training via compute monitoring. arXiv preprint arXiv:2303. 1341, 2023.
[SSK 21] Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems, 84:18021-88 2021.
[TJSP22] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007-4022, 2022.
[TSC 18] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint 181::: 055119, 2018.
[XWL+21] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting trojans using neural analysis. In 202/ IEEE Symposium on Security and Privacy (SP), pages 103-120. IEFF, 2021.
[YZS [2]] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong. Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming- Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint XiXx:2209000996 2022.
[/]L+21]] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint 1.
2021.
[ZLD 22] Rui Zhang, Jian Liu, Yuan Ding, Zhibo Wang, Qingbiao Wu, and Kui Ren. "adversarial examples" for proof-of-learning In 2022 IEEE Symposium Security and Privacy (SP), pages 1408-1422, 2022.
12

---

A Combined Verification Protocol
We can unify the defenses of Section into combined defense protocol, which catches a wide swath of attacks, including all current attacks on from the Proof-of-Leariin literature [FJT 22, ZLD* 22]+
A Prover gives the Verifier transcript T {D, M, W) and a final weight vector W The verifier proceeds to verify whether T is a valid training transcript through the following checks:
1. Check that W ends in the claimed final weights W*.
2. Given the dataset D, hash it to yield the seed in Section 4.3, and use that seed compute the resulting data order II and validation subset D.. (Alternatively, these hashes can be provided by the Prover, and only verified when each point is needed for the protocol.)
3. Check that Wo matches (s). If this fails, reject the transcript.
4. Create an empty list Q to store all suspicious-looking segments to retrain. For each segment W,, II, include it in the list Q to retrain if it fails any of the following checks:
(a) Randomly select
fraction (e.g.. 1% of k) of points 11 from Il, For each such point lli compute the losses on W, and W;-1. shorthanded as sets Lu,, and Ln,,i-1- Similarly, pick an a fraction of points from the validation set D, and compute these points' losses on W, shorthanded LD.. (The validation loss on W,- LD_,s_1, should' ve already been computed when looping on the previous segment.) Also, randomly select ak subset of data points D; from across all training segments 11, and compute these points' losses on W;, CD..s.
If the Verifier wants to plot complete memorization plots, for example as a sanity check or to use in checking for Glue-ing attack as described in Section 6, they can also compute the losses on 28 nearby weight checkpoints W.-g.... -----1 However, this not part of the core protocol, and will not be counted its sample complexity.
(b) Compare the values in LTT. and LD. using the one-sided binomial hypothesis test described in Appendix C, to check that the model used the correct data ordering. If the test cannot reject the null hypothesis that II, does not match W; better than randomly- selected training points D,, then run the test on an even larger fraction of data points from 11, and D to double-check the statistical significance of the null. If this still fails, it suggests that the transcript may have been spoofed. Investigate further (such as by reexecuting training this segment) and/or reject the transcript.
(c) Using Cu, L1,.i-1, LD... and LD.,a-I, calculate a data subtraction upper bound heuristic as in Section 4.2. If this heuristic fails to bound the data subtraction to below an acceptable small limit (which depends on how strongly the Verifier wants catch even small subtraction attacks, and on the Verifier's 's retraining budget), add this segment to Q.
(d) Using (CD... 1 E O..... check that it is smooth at W, and doesn't 't increase decrease discontinuously. If it does, add this segment to Q. (The tolerance should depend on the retraining budget.)
.. .m), compute the mean validation set loss across time, and
(c) Compute the distance I|W; - W:-1ll from the previous checkpoint. Check that this delta similar nearby segments' weight-space deltas. If it is not, add this segment to Q. (The tolerance should depend on the retraining budget. We leave the question of how best to set this threshold, and of below, to minimize false negatives while avoiding reducing false positives that increase retraining costs, to future work.)
5. Randomly sample additional data segments from throughout training, and add them to Q. These additional segments are important establish baseline estimates of segments' weight-space deltas across training, to ensure that there were no systematic data addition attacks at every segment. (Illegal data additions in every segment would shift the entire weight-change delta magnitude plot, thus suppressing anomalies in any one segment).
6. For each segment in the list Q, execute retraining and verify that the resulting weights W; are within an b-ball of the original reported weights W, reported in the transcript.
"To reduce noise when comparing validation performance across checkpoints, this a subset of D, should be the same across all evaluated checkpoints.
13

---

If any values in the re- re-trained weights fail to come within the tolerance that is significant evidence that the transcript has been spoofed, and warrants further investigation. For example, the segment can be retrained more times, to confirm that the weight-space variance across retraining results W(), W2(2) ... is sufficiently smaller than such that the reported W; is clear outlier. If all these tests pass, accept the transcript.
A.1 Complexity
The time costs of training, borne by the the Prover are:
1. h x [D], where h is the cost of a hash, for generating the initial random seed.
2. 8 n, where is the cost of a single gradient computation, and n is the number of training data points.
In comparison, the time costs to the Verifier (assuming the transcript is accepted) are:
1. h |D] hashes for verifying the initial weights.
2. (2 + 1 1) a 5 X n operations for computing the loss of an fraction of datapoints in II, on W; and W and another 2a fraction of points in D, and D We also assume that computing the loss requires 3 the number of operations computing gradient update, which the standard ratio of inference vs. training when using backpropagation.
3. s [Q/m m operations for retraining, where m the total number of checkpoints in the training run.
B Subtraction Upper-Bound Heuristic Derivation
Let W; the checkpoint obtained by training on the full claimed data sequence II, and let W be the checkpoint obtained by excluding s--faction of the points in II; Our goal is to estimate FBQ(I,, ps Wi), the fraction of points in IT, that fall in the bottom p-probability quantile of D, AM values WV:
Let's start with the distribution (m W.), where 11. We make the following two assump- tions:
1. The subtracted s--raction of points are sampled randomly from I1,
2. The p-th percentile of the AM values of D. 's AM values on W; is the same WV:
Firstly, since x- e-frrction of points were subtracted, we know that those subtracted points are distributed like the validation set (assumption 1). Therefore, the fraction of points in II, that lie in our quantile of interest that were subtracted is <FBQ(D,,, p, W!).
Next, we must consider the points that were not subtracted but lie in the quantile. Using assumption we know that (1 œ) fraction of points that were below the quantile remain in the quantile. Therefore, the fraction of points in II, that lie in our quantile of interest that were not subtracted is
(1 - FII,, p, W.).
Putting all together, we have
FBQ(IL, P. W/) xFBQ W/) + +(1 x))LQI p, W:)
There are several things to note about this derivation. First of all, regardless of whether assumption 2 is correct, the second term of Equation 9 is 2 0 and therefore, is still upper-bound. In practice, the actual fraction of subtracted points that lie in the quantile might not be exactly equal to FBQ(D,, P, W?) p. Therefore, if (1 FII p. W,) is very small and close to zero, the resulting upper-bound can be smaller than s, which the case in Figure 6.
Lastly, when computing FBQ values, we can use the unnormalized M values to compute AM as such:
A (d = L(d, W.) - L(d, C(d,W,-) W,-1).
(10)
14

---

This does not affect the FBQ values, since the normalization tenn ERED, [C(d', W)] does not depend on and therefore and has no effect when evaluating A Am(d' W.) > AM W,). Removing the normalization term could represent a 2x efficiency gain. We kept the normalization term in the main paper because it may still be important in other cases for preserving the meaning of the Memorization Delta AM. which is used to produce memorization-chats.. Without the normalization term, it becomes hard to compare A across segments, especially in the earlier stages of training when the changes in loss between checkpoints vary quite a bit across segments.
C Data Order Statistical Test
We want a statistical test that will tell the Verifier whether, for a given training dataset D and data ordering S, which together yield data sequence II, and for given weight checkpoint W., the data segment sequence II; E X explains the memorization pattern of W, better than a random data order/sequence IT; (which
assume is drawn randomly from D). In particular, based on results from Section 4.2, we know that datapoints from the most recent training segment d E II; tend to have higher memorization delta values AM than the average point d' D from the overall training distribution D. Conversely, points from 11; would have no reliably greater AM than the rest of D.
We will use the following test, where II? = II is the null hypothesis and the alternative hypothesis is II' II Let 2 mediande ( W.)). estimated via small number of samples from D. For Pick n, datapoints from data sequence IT? and for each data point TT?, check if it's z. Under the null hypothesis, the probability that each point passes this check is 0.5. Let the test statistic be t:
t(TT')
Σ dI-II' J=
T(A W.) 2)
(11)
where TI is the indicator function. The value of t(TT;), the statistic under the null hypothesis, is distributed as binomial with biased coin probability c = 1/2 and samples. However, we expect that (II,,) is a binomial with a larger c. To compute our confidence that W; was trained using the data order II, we can use one-sided binomial hypothesis test, computing p-value as 1 -1- CDFoommaat (c- = 1/2, ne, k < () where k is the value up to which to calculate the CDR This statistic can be computed jointly across all checkpoints (requiring relatively few samples per checkpoint) to prove that the overall data ordering matches the one defined in Section 4.3.
Note that this test is similar to the "subtraction upper bound" heuristic from Section 4.2. with the key difference being that in this test we compare against the distribution of all training points D (since the counterfactual is randomly selected subset of training data), whereas the subtraction test compares against points from the validation set D, (since the counterfactual is that the points are never included in training). As an additional note, this same test can be generalized by replacing the median with a quantile, which may improve sample efficiency depending on the shape of the AM distribution on II; vs. II/.
15

---

D Verifier Objectives Table
Transcript Case
Attacker Motivation
Definition Defender Sue- cess
Check whether a model W
A Prover wants to claim that W lacks certain ability in or- der to avoid scrutiny, and does by claiming W has only been trained on data from dis- tribution D and not distri- bution D'.
test such that, given tar- get weight checkpoint W, con- firms that its training data did not include, in addition known number of data points from known distribution D. an additional kn training points from different distri- bution D'.
was trained on data from dis-
allowed distribution (c.g., re- lating to backdoors, cyberex- ploit generation, or enabling an undisclosed modality such as images).
Check whether a model W* was trained on greater than certain number of data points, in case policy oversight targets the total training compute of a model (e.g. as part of compute usage reporting).
Underreport total training time to avoid triggering oversight.
test such that, given a tar- get weight checkpoint W and claimed sequence of n data points II, detects whether the model was in fact trained on kn data points, for some k > 1.
Check whether model W was initialized without us- ing weights obtained from previously-trained models.
A Prover might wish training using weights ob- tained from previous train- ing run, hiding the fact that more data or compute was used than reported, in order to avoid scrutiny, or to save compute by copying another's work.
test such that, given desired initialization W, (up hidden unit permutations), makes it cryptographically hard construct transcript that results in Wo being an ini- tialization compatible with the resulting transcript.
Check whether a model has a backdoor, i.c. an improbable input that yields a disallowed behavior.
An attacker might wish to hide capabilities, or give them- selves unauthorized access to systems that will be gatekept by deployed versions of their models.
A test such that, given a transcript, allows reliable de- tection of backdoors through code data audits.
Check whether model was trained using at least certain quantity of data, e.g., as part of Proof-of-Learming meant to verify the original owner of model, or verify that cer- tain safety-best-practice train- ing was done.
A Prover may wish to save on compute costs by doing less training, or to prevent their model from being trained on required data.
A test such that, given a tar- get weight checkpoint W* and a claimed sequence of n data points IT, detects whether the model was in fact trained on en data points, for some e e<l 1.
Check whether model was trained using particular data- point.
A Prover may wish to train on copyrighted content, or un- curated datasets, or obfuscate which training data were used.
A test such that, given tran- script and target datapoint æs detects whether the model was fact trained on æ.
E Hardness of Spoofing a Weight Initialization
To recap, by requiring that the Prover initialize a model's weights specific value in high- dimensional space Wo Rd drawn from pseudorandom vector generator Gr, we seek to disallow class of spoofing attacks based on the Prover hand-picking initial weight vector Wo that will
16

---

after training end up close to Wy, for example by picking an initialization that is already close to W (Attack in [ZLD (2)).
The simplest setting in which defense is impossible, and the Prover can reliably find a random initialization that will converge to given Wy. is in realizable linear models (models with only single linear layer). Since their loss function is strongly convex, any initialization will converge to a neighborhood of the same final value W, making straightforward construct tweaked datasets with certifieddrandom initializations that result in approximately the same final model. Another counterexample occurs when datasets have a single degenerate solution: it possible to construct a 2-layer neural network with training data covering the input space and where all the labels are O, such that the model always converges to a weight vector of all Os, independent of initialization. We will focus our discussion on the usual case of multi-layer NNs with non -on-degenerate solutions, as described below.
Below, we will sketch an informal argument that for some radius r. for a fixed training data sequence 11, the probability that a training run initialized
10 weight vector Wo = (s) ends in a final weight vector W, that is within distance of a particular target vector A. is less than some small value < 5(1 6(1/poly(d)), where d is the dimension of the neural network. This means that a Prover would need to sample a super-polynone find one that would, via training on II, result in a fully-valid training transcript that ends close to the weight vector W, from a previous training run with a different initialization, and therefore that it is exponentially hard violate the "uniqueness" property from Section 3 if the Prover uses a certified random initialization.
geeudoraadommlygeennnza
(in d) number of random seeds to
To understand whether this is the case, of weight initialization, all NNs tend to converge to a small (polynomial) number of modes in weight space. This is indeed the case with linear regression: regardless of the initialization, given sufficient full-rank data all linear model training runs will converge to a neighborhood of the same loss minimum in weight-space. If this were also true for neural networks, then even small number of randomly-sample weight initializations would likely yield at least one weight initialization that, after training, converged
examine the counterfactual claim: that independent
mode close to the target A (assuming which is the case when A is the outcome of a previous training run W Yet, empirically, many works have found that large NNs converge to many different modes [AHS22, FDRC20J.
close to least one mode,
The many modes of the NN loss landscape can be understood through permutation symmetries [AHS22]] Neural networks are equivariant ("equivariant" means that function changes symmet- rically under group action) under specific permutations of their matrices' columns and rows. Nearly all neural networks have the following permutation symmetries: given single hidden layer M2a(M2(D) where M1 E Raxb M2 E O and o R + Re is a nonlinearity, and given any permutation matrix E Rxbb (such that F2 permutes the rows of 2), then by simple algebra Mj (FF
Mia (M2rr for all x. This means that for any set of successive NN matrices M2 there are at least d! possible permutations with identical input output behavior. For a neural network W with A nonlinear layers and hidden dimension of each layer b, there could be k - 1 different permutation matrices F1, F2,.... and we denote to the operation of permuting the flattened weight vector W using a particular value of these I's as P Rdd - Rd. Each P is drawn from the overall set of valid permutations for a particular architecture P E P(M), and we know that [IP(M)) = Ω (2k6
A second important property is that gradient descent is itself equivariant under the described per- mutations. Let be the training operator, such that W, = R(Wo, II) is the result of training initial weights Wo on a data sequence II. II Then it is true that /p E P(M), P(W,) = (R(W,, II) = R(P(W)). II) W?
where W is the result of training on the permuted initialization. This is simply consequence of the fact that the gradient operator commutes with any constant matrix (including the permutation matrix), and that the training process R comprised of repeated calls the gradient operator, additions, and scalar multiplications (both of which also commute with the permutation matrix). 12
10 Assuming that is chosen randomly. based on assumptions described in Section 4.3.
" We omit the inherent noise and hyperparameters inherent in R for brevity.
It is in principle possible to construct optimizers for which this is not the case, but this should hold for all common gradient- grasient-based based NN training optimizers.
17

---

Now, assume that the initialization function G, is radially symmetric (as is the case with all com- mon initialization schemes, e.g., those based on Gaussians), and therefore the probability that the initialization will start at Wo and P(Wo) is the same for all P E P. Then the probabil- ity that the post-training final weights reach W or p(W,) is also the same. If we knew that Pr ( p(W,)) > 2r) > 1 - 8 for some r and small 6, then this derivation would tell us that there are many different weight-space modes into which training could converge, each of which is far apart from the others. (For convenience, let's refer to the number of such far-apart permuted modes as k.)
Again, our goal is to show that a random initialization is unlikely to converge after training to within a neighborhood around some vector A. Assume that B is one of these modes, and ||A B|| r. According to the assumption from the previous paragraph on the distance between post-training modes, for any second mode C, we know that |C B| > 2r with high probability. By the triangle inequality, we know that:
- -
|C - B | BlI
> 2r - rr
Therefore there some minimum distance ||A- C|I > between the target A and all other disjoint modes (each associated with a permutation) of the post-training weight distribution. If the number of such far-apart permutations & superpolynomial, then no polynomial number of weight initialization samples will result in final model close to A.
However, this argument is predicated on sometimes -inetiime--iivlidd assumption: that there are superpolynomially-manl
permutations A = w(poly(d)) of W,. each at least a distance 2r from each other. In the case of the counterexample from the beginning, where all initializations converge after training to the weight vector of all Os, all such permutations are in fact equal, and therefore there no such distance r. Instead, one may need to make an assumption about the non-degeneracy distribution of final weight vectors Wy, such that permutations of these weight vectors are far apart from each other. We leave analysis of which assumptions fulfill this property as future work. Note that for any specific training transcript which includes a specific W,. the distribution of distances of permutations of W can be estimated empirically by manually permuting Wy's matrices.
the
F Experiment Details
For the GPT-2 Experiments we use cosine learning rate schedule that decays by a factor of 10x by the end of training, with linear warmup of 2000 steps to a peak learning rate of 0.0006. For the Pythia evaluation experiments, we choose checkpoints from contiguous blocks out of 144 checkpoints: early (first 19 checkpoints), mid (checkpoints at step 62000 to 80000), and late (last 19 checkpoints).
G More memorization plots
In the following subsections, we plot memorization M, fraction of points with AM above the median, and fraction of points with AM below the 10th percentile. For GPT-2, we use 100% of the data to generate Figures 1, 2, and 3, while for Pythia, we use 10% of the data to generate Figure 2. this section, we show results for smaller sampling rates to highlight that with 1,, or sometimes even 0.1% of the original data, we can still observe the memorization effect.
From Pythia 70M results (Subsections G.1, G.2, and G.3) we can see that as training progresses, memorization effect becomes less pronounced, such that with smaller data sampling rate, less of the diagonal get highlighted (Figures 11, 18, 25), and the histograms are closely overlapping (Figure 32) for the last 18 checkpoints. At the same time, we observe that as the model size increases the memorization effect becomes clearer, even with 0.1 % data sampling rate. In fact, for the 1B- parameter Pythia model, the memorization effect is still clear for the last few checkpoints (Figures 14, 21, 28, and 35) unlike the 70M-paramen case.
II this untrue for all modes B. then by definition there is no initialization that leads close to A, which satisfies our original objective of bounding the probability of the final weights converging to a neighborhood of A.
18

---

G.1
Memorization
GPT-2 120M
Sampling
Sampling Rate
Sampling
Sampling
semment
Mamor Mamorization
segmant
Memorization
segment
Memorizatoo Date segment
0.75
Memorizalin
0.030
11-
0.02
0
0.000
17
Checkpoint
Figure Memorization plots for GPT-2 with different sampling rates.
Pythia 70M Early
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
25
380 segment
Data segment
segment
Nemortzalln
110
17
17-
17
Checkpoint Num.
Checkpaint
Checkpaint
Figure 9: Memorization plots for the first 18 checkpoints of Pythia (70M) with different sampling rates.
Pythia 70M Mid
Sampling Rate 0.1
Sampling Rate 0.01
Samoling Rate 0.001
oata Segment
15
Memorization
segment
Memorization
segment
1.0
17
17-
17
de 72 74 76 7 Checkpoint Num.
Checkpoint
Checkpaint
Figure 10: Memorization plots for checkpoints near the middle of Pythia (70M) with different sampling rates.
Pythia 70M Late
Sampling Rate
x10
Sampling Rate 0.01
Sampling Rate 0.001
x30
1.0
Data segment
0.8
Data segment
memorizttion
segment
13
13
-1.0
13
1.5
125
128 131 19c
137 140 143 Checkpoint Num.
125
131 134
140
143
128
131
137
140 143
Checkpoint Num.
Checkpoint Num.
Figure 11: Memorization plots for the last checkpoints of Pythia (70M) with different sampling rates.
19

---

Pythia 410M Mid
Sampling Rate 0.1
x10-1
Sampling Rate 0.01
Sampling Rate 0.001
x30-1
Data segient
Memo Cemorizatioo
Data seement
Momortbation
12
11
62 64 dé 66 TO 72 74 76 Checkpoint
64 66
16
64
15 Checkpoint
Checkpoint
Figure 12: Memorization plots for checkpoints near the middle of Pythia (410M) with different sampling rates.
Pythia 1B Mid
Sampling Rate 0.1
Sampling Rate 0.01
Samsling Rate 0.001
x10
Data segment
nemorizatonn
Data segment
Momorizatiin
11
02 64 66 68 74 76
78
80
62 64 66 08 70 72 Checkpaint
64
Checkpoint
Figure 13: Memorization plots for checkpoints near the middle of Pythia (1B) with different sampling rates.
Pythia Late
Sampling Rate
-10-1
Sampling
w10
Sampling Rate 0.001
x30-
25
20
Data segmennt
Data sagment
Memoriza too
Data segment
-55
7-
1.0
16.
16
16-
0.5
125 128 131 134 137 140 143
125
128
131 Checkpoint Num.
151 197 140 143
125
128
197 140 Checkpoint Num.
131 134
Checkpoint Num.
Figure 14: Memorization plots for checkpoints near the end of Pythia (1B) training with different sampling rates.
20

---

G.2
Fraction of Samples Above 50th Percentile
GPT-2 120M
Sampling Rate 1.0
above parcenttee sagreent
Sampling Rate 0.1
Sampling Rate 0.01
Frec: abovv S/0t per percentle
Sampling g Rate 0.001
Free. above percentle
Data segment
Data segment
tr d F
tr t 1
13-
15
Checkpoint Num.
Checkpoint Num.
Checkpoint Num.
Checkpoint Num.
Figure 15: Fraction of samples above the 50th percentile for GPT-2 with different sampling rates.
Pythia 70M Early
Sampling
Sampling
Sampling 0.001
3.8
Frac above 50th percentie
0.
Frac. 5.0t percenticc
0.8
0.7
Data segment
Data segment
segment
-0.6
11:
11
0.
11
13:
13
15
9.3
17
92
13 15 17
5
7
Checkpoint Num.
Checkpoint Num.
Checkpoint Num.
Figure 16: Fraction samples above the 50th percentile for the first 18 checkpoints Pythia (70M) with different sampling rates.
Pythia 70M Mid
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0, 0,001
0.65
Faac. above 50th percentie
0.63
Frcc. percentlle
0.00
Oata segment
segment
segment
23.
0.550
11
0.50
11
-0.50
13-
13
13
0.45
15
045
15
17
17
0.40
63 65 67 69 71 Checkpoint Num.
05
67
73 75 77 79 Checkpoint Num.
69
72,
71 73 75
77 79
Checkpoint Num.
Figure 17: Fraction of samples above the 50th percentile for Pythia (70M) with different sampling rates.
Pythia 70M Late
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
0.515
6.52
Data segment
0.510
Data segment
Data segment
0.505
20
0..5c
3,500
0.48
13.
13
0.455
126 132 135
238 141
126
132
139 141
126 132
139
141
Checkpoint Num.
Checkpoint Num.
Checkpoint Num.
Figure 18: Fraction samples above the 50th percentile for the last 18 checkpoints Pythia (70M) with different sampling rates.
21

---

Pythia 410M Mid
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
Frac. above 50th percentlle
above percentile
-0.3
Data segment
Data segment
segment
-0.6
06
-.8
-0.4
04
-344
22
82
63 65 67 ri 15 79 Checkpaint
67 6s
71
15
67 69 71 73 Checkpoint
Figure 19: Fraction of samples above the 50th percentile for Pythia (410M) with different sampling rates.
Pythia 1B Mid Sampling Rate 0.01
Sampling Rate
Sampling Rate 0.001
1.0
Frac: above 50th percentie
percentie
1..8
0.8
-.88
Data segment
Data segment
-94
12
0.4
32
22
82
63 05
75 79
05
G7
67 09
Checkpaint Num.
Checkpoint Num.
Checkpoint Num.
Figure 20: Fraction of samples above the 50th percentile for Pythia (1B) with different sampling rates.
Pythia 1B Late
Sampling
Sampling
Sampling Rate 0.001
1.0
0.
Frat above 50th percentie
0.
Frat. above 50th percenttle
0.9
3.8
0.
0.8
Data segmient
Data segmient
segment
13.
0.4
0.4
133
0.4
03
03
Fec
16.
16
16
92
126
129
132
135
138
141
126
129
132
135
13B
142
126
129
132
135
139 141
Checkpoint Num.
Checkpoint Num.
Checkpoint Num.
Figure 21: Fraction of samples above the 50th percentile for checkpoints near the end of Pythia (1B)
training with different sampling rates.
22

---

G.3 Fraction of Samples Below 10th Percentile
GPT-2 120M
Sampling Rate
Frac below 10th percentle
Sampling Rate
Frac below 10th percentle
Sampling Rate 0.01
Frac. bekow 10th percentle
Sampling Rate 0.001
Frec. percentlee
segieen
Data segirant
sesment
20
seament t
sn
10
10
13
13-
10
20
2014
Checkgoint
Checkpoint
Checkpoint
Checkpoint
Figure Fraction of samples below 10th percentile GPT-2 with different sampling rates. White
boxes occur whenever the number of samples falling below the 10th percentile is O.
Pythia 70M Early
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0,001
Frac. below 10th percenale
Frac. below 10th percentiee
Data segmert
Data segmert
7-
7-
10
110-
10-1
10th
11
Data
100
7 9 Checkpeint Num.
5
11 13 15 17
3
11 13 15 17 Checkpoint Num.
5
7 5
11 13 15 17 Checkpoint Num.
Figure 23: Fraction of samples below 10th percentile for the first 18 checkpoints of Pythia (70M)
with different sampling rates. White boxes occur whenever the number samples falling below the 10th percentile is 0.
Pythia 70M Mid
Sampling
Sampling
Sampling
below 10hh percentle
frac. below 10th percentle
percentile
Rala seghrent
sogment
Data segment
7
9-
10
9-
10rl
9
-1011
21.
11
11.
13
15
63 65 67 69 75 77 79 Checkpoint Num.
63.
89 71 73 75 77 Checkpaint Num.
63 65 67 69 V. Chockpoint Num.
Figure 24: Fraction samples below 10th percentile for Pythia (70M) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
23

---

Pythia 70M Late
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
1.4
Frai: below 10th percentlle
Frac. below 10th percentlle
percentille
10X
105
Data segment
Data segment
segment
10s
10-1
20-
08x
10-
10
Maxten
16
16-
16-
A3
126 129 133 135 158 Checkpoint Num.
126
129 132 195 138 141
128 129 132 135 138 141 Checkpoint Num.
Checkpoint Num.
Figure 25: Fraction of samples below 10th percentile for the last 18 checkpoints of Pythia (70M) with different sampling rates.
Pythia 410M Mid
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
Frat. below 10th percentie
Frac. below 10th percentle
Osta segment
segment
21-
19th
11-
Data
11
23-
13
13-
15.
10-
15-
17
17-
17-
10
75 Checkpaint Num.
Chackpaint Num.
Checkpoint Num.
Figure 26: Fraction of samples below 10th percentile for Pythia (410M) with different sampling rates.
White boxes occur whenever the number of samples falling below the 10th percentile is O.
Pythia 1B Mid
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0,001
Frac. below 10th percentic
Frac. below 10th percentile
Data segment
10-1
Data segment
10-l
8-
9-
30-4
21.
-10
11
Data
11
Frac. 10-
13
13
15.
15-
10
15-
17-
17-
63 65 67
71 75 77 Checkpaint Num.
69
65
67
69 71 73 77 Checkpoint Num.
Checkpoint Num.
Figure 27: Fraction of samples below 10th percentile for Pythia (1B) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is O.

---

Pythia Late
Sampling Rate 0.1
Sampling Rate 0.01
Sampling Rate 0.001
Frac. below 10th parcemile
Frec. below 10th percentile
Data segment
Data segment
Data segment
110
10
20-
20-
10-
16
16-
16-
126 129 132
138 141
126
129
192 Checkpoint
135
139
129 132
15s
138 141
Checkpoint
Figure 28: Fraction of samples below 10th percentile for checkpoints near the end of Pythia (1B) training with different sampling rates. White boxes occur whenever the number samples falling below the 10th percentile is O.
25

---

G.4
Memorization Delta Histograms
GPT-2 120M
Chackpaint
Checkpoint
Chackpoint
Chockpoint 444
1.4
1.2
man vals.
3.5
3.0
ser valdd wut
vol d.s.
min ret
12-
nan set
vo
1.0
Density
0.6
044
2
0.0
-0.4
-0.2
0.9
3-2
2.4
-0.4
-0-2
5.0
0.2
0.4
-34
-0.2
0.5
0.2
0.4
-2.4
-9.2
0.0
022
0.4
Memorization Delta
Memorization Deña
Memorization Delta
Momarization Delta
Checkpoint
Checkpoint #6
Checkpoint #?
Checkpoint +8
Ian
15.0
sar
vali stt
17.5
sol
min set
17.5
12.5
Dansity
Denaity
10.0
Denslty
10.0
7.5
5.0
2.5
0.0
-0.4
-0.2
0.9
5.2
34
-0.4
-0-2
5.0
0.2
0.4
-244
-0.2
0,5
0.2
04
-244
-92
0.0
02
0.4
Memorization Delta
Memorization Deka
Memorization Delta
Memarization Delta
Chackpaint #9
Checkpoint #10
Cherkpaint #11
Checkpoint #12
Irsh val
tan
vadi st
20
val
tas set
20
Dansaty
Density
Density
Denslty
10
0.4
-02
0.0
5.2
0.4
0..4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
02
0.4
Memorization
Memorization
Memorization
Momarization Dalta
Checkpoint 413
Checkpoint #14
Cherkpaint
#15
Checkpaint #16
25
tran set
valc.
vald.
1ss1 set
35
wa
na
25
Denstt
Dersity
Density
20
Density
25-
15
20-
10
15
10
522
-0.2
0.4
-3.4
-0.2
0.0
0.2
DA
-2.1
-3.2
0.0
0.2
0.4
Mamorization Delta
Memorization Deka
Momorization Delta
Momarization Delta
Checkpoint #17
bon stt
vako.
Denaity
20
Memorication
Figure 29: Each subplot compares two histograms of A one for AM resulting from the checkpoint evaluated its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from GPT-2 training.
26

---

Pythia 70M Early
Cheikt e1
Checkpoint #2
Checkpoint #3
Checkpoint
0.7
12
17.5
0.6
15.0
0.5
Densiyy
Densfty
12.5
0.4
0.3
0.2
0.3
2.5
0.0
-1
-0.4
-0.2
0,0
0.2
-0.4
-02
0.0
0.2
D4
0.0
-0.4
-0.2
0.2
D.4
Memorization Delta
Memorization Delta
Memorization Delta
Memorization Delta
Check knaint
Checkpcint
Checkpoint
Checkpoint
net
ni
10h
val. aut
20
va1s
20
20
Denaity
Danilty
155
Dansity
Dansity
L5
20
10
0.4
0.2
0.0
0.2
0.4
0.4
6.2
0.0
0.2
0.
0.4
0.2
6.0
0.2
04
0.4
0.2
0.0
0.2
0.4
Memoribation Delta
Memorization Delta
Memortzation
Memorizattion Dalta
Checkpoint #9
Checkpoint #10
Checkpoint #11
Checkpoint #12
matt
25
25
vanae
20
20
20
20-
Denaiyy
Denoty
15
15
10
10-
0.5
-5.4
Memorization Delta
Memorization Delta
Memonization Delta
Memorization Delta
Checkpoint #13
Checkpoint #14
Checkpoint 11
Checkpoint #16
25
25
25
d.
vi
20
20
Dendit
Derroity
3.
10
-0.4
-0.2
0.9
0.2
0.4
-0.4
-0.2
0.0
0.2
-0.4
-02
0.0
02
0.4
-0.4
-0.2
0.0
0.2
0.4
Memorization Delta
Memoribation Delta
Memorization Delta
Memorization Delt
Charkpaint #17
Checkpaint #18
taan vol. uut
1i vot
20
20
Denttty
Deietty
3.5
15
0.4
0.2 Momorization Delta
-.2
0.0
0.4
-6.2
0.0 0.2
0.4
Momorization
Figure 30: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are the first 18 from Pythia (70M) training.
27

---

Pythia 70M Mid
Checkgoint #63
Checkpaint #64
Checkpoint #65
Checkpoint w66
10
Deenlit
Dursfty
Density
Denait
2-
-9.4
-0.2
0.0
02
-0.4
-3.2
0.0
0.4
-0.2
0.0
0.4
-5.4
-0.2
0.0
0.2
D.
Memorization Delta
Memorication Delt
Memoriattio Delta
Memorization Deka
Checkpaint #67
Checkpaint #68
Chackpaint #69
Checkpoint #70
vor
tons
seanset
19h 901
vals, set
vald. set
vald set
va1s. s0:
Dendit
Danaity
Danaity
Density
1-
0.4
0.2
0.0
5.2
0.4
0.4
5.2
0.0
0.2
0.4
0.4
0.2
8.0
0.2
0.4
0.4
0.2
0.5
0.2
0.4
Memortzation
Mimorization Delta
Momorization Delta
Marnorization Daña
Checkpoint #71
Checkpalit nt #72
Checkpoint
#73
Checkpoint #74
bur vd
bua
Jun 94
Wil.
vol
Dendiy
Density
Dendtty
Density
1
2.6
0.1
Memorization
Memorization
Memorization Delta Checkpoint +77
Memonization Deta
Checkgoint #75
Checkpaint #76
Checkpoint #78
vais,
wall.
Denslty
Censity
Deriity
Denaity
-0.4
-0.2
0.0
0.2
0.4
-0.4
-32
0.0
0.4
-0.4
-0.2
0.0
0.2
0.4
-2.4
-0.2
0.0
0.2
0.4
Memorization Delta
Memorization Delia
Meroriizttion Delta
Memarization Deka
Checkpaint #79
Chockpai nt #80
vrun vels. 24
wald.a0 act
Deniity
Denuity
0.44
-0.2 Momortzation
0.0
5.2
0.4
0.2 Momorization Delta
-.22
0.0
0.4
Figure 31: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (70M) training.
28

---

Pythia 70M Late
Checkpoint #126
Checkpoint 1+2
Checkpoint #128
Checkpoint #129
3.0
3.5
2.5
Density
Dirisity
1.5
1.5
1.0
1.0
0.5
0.5
0.0
-0.4
0.2 Memorization Delts
-02
0.0
-0.4
-2.2
0.5
0.2
04
-0.4
-0.2
0.0
0.2
0.4
0.0
-9.4
-0.2
0.0
0.2
0.4
Memonization Delta
Memorisation Delta
Memorizatin Delta
Checkpoint +130
Checkpoint #131
Chackpaint #132
Chackpoint #133
1011
tran set
10t 901
vall. 141
val set
va15 s0
Densit
Dansity
Density
Dansity
0.4
0.2
0.0
0.2
0.4
0.4
6.2
0.0
0.2
0.4
0.4
6.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
04
Momorization Delta
Mamorization Delta
Mamorization Dalta
Memorization Delta
Checkpoint #134
Checkpoint #135
Checkpoint #136
Checkpoint #137
3.5
Inet sel
34
buisset
val
Jun 90
3.0
Demity
Density
Denstyy
2.0
0.5
0.0
0.2
Memorization
Memorization Checkpoint 139
Momonisation Checkpoint #140
Memorization Delta
Checkpoint +138
Checkpoint #141
3.5
3.0
Denatty
Density
Density
2.0
2
1.3
1.0
0.5
-0.4
-0.2
0.0
0.2 0.4
-0.4
-0.2
0.0
0.2
0.4
-0.4
-0.2
0.0
0.2
0.4
0.0
-0.4
-0.2
0.0
0.2
0A
Memorization Delta
Memorization Delta
Memarizatian Delta
Memorization Delta
Checkpoint #142
Checkpoint #143
tnst vall. 14:
35
3.0
Densty
1.5
1.0
0.5
0.4
-0.2
9.0
0.2
0.4
-0.4
-6.2
0.0
0.2
0.4
Memorization
Mamorization
Figure 32: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are the last 18 from Pythia (70M) training.
29

---

Pythia 410M Mid
Checkpoint #63
Checkpoint v64
Checkpoint #65
Checkpoint #66
35
30-
15
15-
10
10
10
5
-244
-0.2
3.0
02
0.4
-0.4
-0.2
0.5
0.2
D4
-0.4
-0.2
0.2
0.4
-0.4
-0.2
0.0
02
D.4
Memoriztiin Delta
Memonization Delta
Memonization Delta
Memorization Delts
Chockpoint
Charkpoint 468
Checkpoint
Checkpaint
35
Irah
vall. set
vo net
val set
vass se:
25
Dencity
20
0.4
0.2
5.0
02
0.4
0.4
0.2
0.5
0.2
0.
0.4
0.2
0.0
0.2
0.4
0.4
6.2
0.0
0.2
0.4
Memorization
Memorization Delta
Mamarization Dalta
Memorization Delta
Checkpoint #71
Checkpoint #72
Checkpoint
#:73
Checkpoint #74
Intt sel
15
zan 94
5
val
Density
Denstyy
02
Memorization
Memorization Delta
Memorization Delta Checkpoint #77
Memorization Delta
Checkpoint #75
Checkpoint #76
Checkpoint #78
25
30
25
Deniity
20-
15
12
10-
12
10
10
5
-2.4
-0.2
2.0
02
0.4
-0.4
-0.2
0.5
0.2
04
-0.4
-0.2
0.2
0.4
-04
-0.2
0.0
0.2
0.4
Memorization Delta
Memorization Delta
Memoriiattion Delta
Memorization Delta
Chackpoint
Chackpoint
35
anst vall. 141
30
Denuity
20
1.5
1.-
-0.4
-0.2
3.0
02
0.4
-0.4
0.2 Memorization Dalta
-0.2
0.0
0.4
Memorization
Figure 33: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (410M) training.
30

---

Pythia 1B Mid
Checkpoint #6
Checkpoint v64
Checkpoint #65
Checkpoint #66
35
mik
30
30
25
25
2.0
20
20
15
15
15
10-
10
10
10
-24
-0.2
3.0
02
044
-0.4
-0.2
0.5
0.2
04
-0.4
-0.2
0.2
0.4
-0.4
-0.2
0.0
02
D.4
Memoriztiin Delta
Memonization Delta
Memonization Delta
Memorization Delts
Chockpoint
Charkpoint 468
Checkpoint
Checkpaint
train set
vall. s4t
1o not
val set
vass se:
30
Density
Denuity
Dencity
20
20
25-
25
25-
10
10-
10
10-
0.4
0.2
9.0
02
0.4
0.4
0.2
0.5
0.2
04
0.4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
0.4
Memorization
Memorization Delta
Mararizaatin Dalta
Memorization Delta
Checkpoint
#71
Checkpoint #72
Checkpoint #:73
Checkpoint #74
Innt sel
40
Van 94
val
30
Densty
Densiyy
Densiyy
20
20
20
10
10-
02
Memorization
Memorization Delta
Memorization Dett Checkpoint #77
Memorization Delta
Checkpoint #75
Checkpoint #76
Checkpoint #78
40
40
vald
35
20
Dvesity
Dersity
Dencity
20
22
-2.4
-0.2
2.0
02
0.4
-0.4
-0.2
0.5
0.2
04
-0.4
-0.2
0.2
0.4
-04
-0.2
0.0
02
0.4
Memorization Delta
Memorization Delta
Memorizatiin Delta
Memorization Delta
Chackpoint
Chackpoint
40
antt
vall. 14:
30
Deniiyy
Denuity
10
-0.4
-0.2
3.0
02
0.4
-0.4
0.2 Memorization Dalta
-0.2
0.0
0.4
Momorization
Figure 34: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (IB) training.
31

---

Pythia 1B Late
Checkpoint #126
Checkgoint #127
Checkpoint #128
Checkpoint #129
70
70-
60-
50
50
40
4.
40
40
30
10
10-
10
-244
-0.2
3.0
02
0.4
-0.4
-0.2
0.2
-0.4
-0.2
0.2
0.4
-044
-0.2
0.0
0.2
D.4
Memorizttion Delta
Meronization Delta
Memorization Delta
Memorization Delts
Checkpaint #1330
Checkpoint #131
Checkpaint #132
Checkpoint #133
70
train
60
vall. se:
60
1o not
val set
vass
50
40-
40
30
-30
30
23
20
20-
0.4
0.2
9.0
02
0.4
0.4
0.2
0.0
0.2
0.
0.4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
0.4
Memorization
Memorization Delta
Mamarizatian Dalta
Memorization Delta
Checkpoint #134
Checkpoint #135
Checkpoint #136
Checkpoint #137
70
Init sel
70
70
zanse 94:
60
val
50
40
Denstyy
-22
30
20
20
20
10-
10
10
10-
02
Memorization
Memorication Delta
Mamonization Deta
Memorization Delta
Checkgoint #138
Checkgoint #139
Checkpoint #140
Checkpoint #141
70-
73
60
60
60
vald
50
40
30
30
10
-2.4
-0.2
2.0
02
0.4
-0.4
-0.2
0.5
0.2
04
-0.4
-0.2
0.2
0.4
-044
-0.2
0.0
022
0.4
Memorization Delta
Memorization Delta
Memorizattion Delta
Memorization Delta
Chackpaint #142
Checkpaint
60
voll. 141
50
30
20
10
-0.4
-0.2
3.0
02
0.4
-0.4
0.2 Memorization Dalta
-0.2
0.0
0.4
Memorization
Figure 35: Each subplot compares two histograms of A.. one for AM resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the end of Pythia (1 (IB) training.
32

---

H
More attack plots
H.1
Interpolation Attack
We repeat the interpolation attack experiment in the main body of the paper, with the IB 1B-parameter Pythia model and observe from Figure 36 that indeed the interpolated checkpoints fail our memoriza- tion tests.
Memorization
X10
oa Ran
Interonated Run
Validation Loss
2.185
Segment
5
2.180
9-
2.170
0
15
2.165
17
4 8 12 Checkpoint Num.
Checkpoint Num.
Figure 36: Simulating an interpolation attack by training a Pythia (IB) model until the 67th checkpoint, to the 80th checkpoint. On the left, we show that an attacker can and then interpolation points to mask any irregularities in validation loss. (The green line carefully choose perfectly overlaps with the blue line.) Nonetheless, on the right, memorization plot, computed using only 1% of data: the typical memorization pattern along the diagonal does not exist for the interpolated checkpoints. For each row corresponding to a data segment II; a box marks the maximal-M checkpoint. The box is red if the checkpoint is a match W;, and magenta if there is no match and the test fails W 1/16
clear signature in the
H.2 Data Subtraction Attack Tests
In the following subsections, we plot the subtraction-upper-buum heuristic A(I1; p. W;) with varying values of p. for different subtraction rates. We observe that for big enough models À is tight upper-bound when no subtraction has happened. For Pythia with 70M parameters, our smallest model, À does not provide tight upper-bound However, for GPT-2 with 124M parameters, Pythia with 410M parameters, and Pythia with IB parameters, À provides a tight boper-
For the IB- B--paamettrr Pythia model, we further plot the upper-bound bound heuristic for varying values of the checkpoint interval (number of training steps between each checkpoint). From Figure 41, we observe that even though À increases as the interval increases, it is still a good upper-bound (~0..05 for a checkpoint interval of 5000 steps) for p 0.1 and = 0.2. This means that we can save checkpoints less frequently and still use the heuristic to detect data subtraction.
H.2.1 GPT-2
hut Data
sustacteen
50% Subrractionn
65% Sustractor
1.25
1.25
1.25
3.25
2000
1.00
1.00
3.07
0.75
7.
0.75
0.77
0.5
5.50
100
2.50
0.5
05
2.25
2.5
0.0
0000
2.00
10
10
Checkpoiit
Checppon
Figure 37: The dubrraction-uppe--bbund bound heuristic A(I,, p, Wt) is robust to different values of p. especially for the full data case. The heuristic was computed using just 1% of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
33

---

H.2.2 Pythia (70M)
Full Data
5%5 Subtraction
50% Subtraction
Saberatton Lawerooundd
1.30
1.50
1.50
1.25
125
1:25
p 0.01
1.00
1.00
0.75
0.75
0.75
0-
0.50
0.50
0.50
P-04
0.00
65
70
75
00
85
70
75
Checkpoint Num.
Checkpoimt Num.
Checkpaint Num.
Figure 38: The dubtraction-upper-boot for a smaller model (Pythia with 70M parameters). The heuristic was computed using 1% training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
heuristic X(II, p, W,) does not provide good upper bound
H.2.3 Pythia (410M)
Full Data
5% Subtraction
50% Subtracion
Subtraction Lowsr-liuune
p= 0.01
0.6
0.6
0.6
0 0.2
0.4
0.4
0.4
p 0
65
75
80
65
70
75
80
65
70
75
Chackpoint Num.
Checkipoint Num.
Checkocint Num.
Figure 39: The subtraction-uppet ceppetioonn-ppee-bouud heuristic X(II, p. Ws) is good upper bound for big enough model (Pythia with 410M parameters). The heuristic was computed using 1% of training data, across 20 random seeds, with dashed lines showing the subtraction rate.
H.2.4 Pythia (1B)
Nubtraction Low.er-boore Aee---rundd
0.8
0.8
0.8
p= 0,01
0.4
0.4
0.4
0.0
Checkpoint Num.
Checkpoint Num.
Checkscint Num.
heuristic X(II, p. W;) is a good upper bound for bigger model (Pythia with 1B parameters). The heuristic was computed using 1% of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
Figure 40: The subtraction
34

---

000
E.9
2.0
000
Chelpprr
Figure 41: Even though the subtraction-uppe-bbuure heuristic A(II, P. W:) increases as number of steps between checkpoints increases, it still provides a good upper bound for 0.1 and p 0.2. The heuristic was computed using 1% of training data, across random seeds, with dashed lines showing the true subtraction rate.
I Broader Impacts
We intend this work to be a step towards meaningful and transparent public oversight of large Al systems, especially those with capabilities whose irresponsible could significantly harm the public. Our protocol is sketch of technical framework for a system by which AI developers can prove properties of their training data, and may thereby enable the effective enforcement of broader set of policies than those solely relying on querying models "black-box". While enabling many possible positive rules, this could also be misused by coercive states to detect and enforce harmful restrictions on beneficial AI development. However, in most cases, such authoritarian states would already have means for policing domestic Al developers' behavior, and verification tools demanding so much cooperation from the Prover are unlikely to meaningfully increase existing surveillance powers. Another issue is that requirements for complying with monitoring and enforcement tend to favor large companies, for whom the cost of compliance can more easily be amortized. This motivates efforts to keep verification schemes simple, flexible and cheap.
We hope that this protocol can also be useful for verifying agreements between untrusting countries. The protocol itself does not provide means for identifying that Al model was developed in the first place unless it is disclosed. In this sense, it more closely parallels a process for an Al-developing country allow its counterpart to retroactively inspecting a developed system (paralleling the New START treaties' inspections of nuclear launchers). rather than to proactively detect when a new system is deveoped (paralleling the IAEA's monitoring of the process of uranium enrichment).
Because our protocol supports multiple independent auditors reviewing the same transcripts, we hope that these tools will support the development of trust between competing companies and countries. Ultimately we hope such protocols will support the development of a larger governance ecosystem representing many parties.
35


Reviews:
[{"review_id": "GI8jREeg9B", "reviewer": "Reviewer_Ejfn", "summary": "This paper describes techniques and tools that can be used for verifying the \"provenance\" of large neural models, to evaluate their risks. These techniques and tools are part of \"protocols\" used by a model trainer to convince a \"verifier\" that the training data was used to produce the model parameters. The authors show experimentally that their prescribed procedures can catch a variety of known attacks from the \"proof-of-learning\" literature.", "questions": "Please see above.", "limitations": "In conclusion this is very interesting work but may be too preliminary for publication.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper addresses an increasingly important problem, as large neural models are becoming very popular, and advances practical techniques that can be used by regulators to check the provenance of large models.\n\n* The authors present convincing evaluation using GPT-2, demonstrating that the proposed procedures are effective in catching a variety of attacks, such as glue-ing and interpolation as well as data addition and subtraction.", "weaknesses": "* I find that the use of \"proofs\" in the title and throughout the paper is misleading as the authors do not present techniques that amount to an actual proof.\n\n* I think it is great that the techniques presented by the authors are practical and effective wrt several attacks but I wonder if this is enough for regulators. I mean they would possibly need stronger guarantees for such techniques. \n\n* It is unclear to me how the verification strategies presented in section 3 relate to definition 1. The authors should work on adding a theorem that clearly states that their strategies achieve the desired properties, i.e., the verifier accepts/rejects true winesses/spoofs with the desired probabilities.\n\n* The technical contribution beyond \"proof-of-learning\" is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Below, we respond specific points made in your review:\n\n**W1:** *I find that the use of \"proofs\" in the title and throughout the paper is misleading as the authors do not present techniques that amount to an actual proof.*\n\n**Response:** The use of the word “Proof” in Proof-of-Training-Data comes from a large body of existing literature (e.g. “Proofs-of-Learning”, which are themselves also not “proofs”).  However, we agree that this may be confusing, so we have changed the paper’s title for the final version to “Tools for Verifying Neural Models’ Training Data”.\n\n**W2:** *I think it is great that the techniques presented by the authors are practical and effective wrt several attacks but I wonder if this is enough for regulators. I mean they would possibly need stronger guarantees for such techniques.*\n\n**Response:** In developing this work, we spoke with US regulators who expressed interest in using these techniques, as they substantially improve the state of verifiability relative to the current baseline (which is blindly trusting Provers). Indeed, much real-world regulatory verification is done heuristically rather than with proofs.\n\n**W3:** *It is unclear to me how the verification strategies presented in section 3 relate to definition 1. The authors should work on adding a theorem that clearly states that their strategies achieve the desired properties, i.e., the verifier accepts/rejects true winesses/spoofs with the desired probabilities.*\n\n**Response:** Thank you for bringing this to our attention. We will add a subsection in Section 2 better clarifying Definition 1’s connection to our contribution, summarized below:\n- Definition 1 is intended to formalize the PoTD problem and defines robustness as being over all computable adversaries $\\mathcal{A}$, but mathematically proving such guarantees is not yet possible given the lagging state of the NN theory literature. Indeed Jia et al.’s PoL protocol [1] itself does not make claims to provable robustness, as evidenced by attacks that have been found.\n- Instead, we follow the heuristic approach common in the PoL literature by enumerating a common-sense list of possible attacks, which provides a first approximation to the full space of attacks $\\mathcal{A}$ in the definition.\n- We then show a solution that is robust to these attacks, and thereby demonstrate that our defenses heuristically satisfy Definition 1. As mentioned in lines 283-291, we hope future work will find new attacks (thereby better approximating $\\mathcal{A}$) which could need to be addressed with new methods, just as the original PoL paper inspired further attacks and defenses.\n\n[1] Jia, Hengrui, et al. \"Proof-of-learning: Definitions and practice.\" 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.\n\n**W4:** *The technical contribution beyond \"proof-of-learning\" is unclear.*\n\n**Response:** To clarify, beyond defining the new problem of Proof-of-Training-Data (which is distinct from the existing PoL problem), we also contribute the following:\n- We propose two new defenses, memorization-tests and verifiable data shuffling, which successfully address all published (and currently unaddressed) attacks on PoL. We also highlight several new attacks specific to PoTD (data addition & subtraction) and show that our methods can be effective at catching these too.\n- Our methods are substantially more efficient than prior work, which means that we are also the first to scale PoL/PoTD methods to LLM-scale using academic compute budgets. (Prior PoL papers scaled only to ResNet-50 on CIFAR.)\n"}, {"review_id": "nZOhScDM5F", "reviewer": "Reviewer_yjkq", "summary": "The paper presents a novel protocol called Proof-of-Training-Data, which a third party auditor can verify the data used to train a model. Here, the auditor will require training data, training code, and intermediate checkpoints. Experiments on two language models have demonstrated that known attacks from the Proof-of-Learning literature can be caught by this new protocol.", "questions": "- Line 77, could there be a comparison between the formal formulation of Proof-of-Learning and Proof-of-Training-Data?\n- Section 4, as most of the Proof-of-Learning experiments have been done on image datasets like CIFAR, I am curious if PoTD could perform similarly well on those image datasets.\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper attempts to tackle an important security problem on trained neural network models. The proposed heuristics (i.e., memorization-based tests) is appealing and can efficiently catch spoofed checkpoints using a small amount of data. The paper is adequately structured and solid experiments have been carried out to empirically justify the effectiveness of the proposed protocol.", "weaknesses": "The concept of Proof-of-Learning has been well studied. Although the authors have discussed various Proof-of-Learning literature in the related work and the experiments section, it is still not immediately clear to me why we need this brand-new protocol (Proof-of-Training-Data). If I understand correctly, the authors are attempting to solve an even harder problem where the adversaries can have more computing power. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Below, we respond specific points made in your review:\n\n**W1:** *It is still not immediately clear to me why we need this brand-new protocol (Proof-of-Training-Data).*\n\n**Response:** We were uncertain as to whether you were unclear about “the difference between the definitions of PoL and PoTD”, or unclear about “why existing methods from the PoL literature are insufficient to guarantee PoTD”. Just in case, we provide answers to both.\n\nFirst, we clarify the difference between PoL and PoTD. **PoL only checks if the Prover is capable of spending the compute** to perform a large training run, but does not check whether the training transcript disclosed **actually corresponds** to the training run that yielded $W^*$. For example, a Prover can do an original training run, and then replace 10% of the original training data with synthetic data as in [1] (e.g. to hide that data from the Verifier), and the resulting modified transcript **would still be a valid PoL.** PoTD is more ambitious in that it **checks whether the exact reported data transcript would actually result in** $W^*$. We will better highlight this distinction in the camera ready.\n\nNext, we clarify why existing methods from the PoL literature are insufficient to solve our PoTD problem:\n- **Segment-wise retraining is known to be potentially vulnerable to synthetic data attacks** [1], which allow the Prover to swap the real data out for carefully-crafted fake data. Our new verified-data-shuffling defense in Section 3.3 stops this attack.\n- **Retraining is inefficient for catching small data lies**. Given 100 checkpoints, if a Prover wants to misreport 1 checkpoint’s worth of data, then to catch them w/ 80% probability the Verifier would have to rerun 80% of the original training run. However, with our method, the Verifier needs to spend as little as 1% (and only inference compute) to catch subtraction/interpolation attacks, and slightly more for addition attacks (which require retraining a couple segments to calibrate segment-magnitudes, but still yield a roughly order-of-magnitude improvement). Even if a Verifier is only worried about Provers misreporting 10 checkpoints, the Verifier would need to retrain $\\approx 16\\%$ of the checkpoints (see Appendix C of [2] for a similar derivation) to catch the lying Prover with 80% probability, much larger than the 1% or few percent required by our method.\n\nWe will make this direct comparison between PoL and PoTD clearer in the final version of the paper, and use the above examples to better highlight the shortcomings of prior methods at achieving PoTD.\n\n[1] Zhang, Rui, et al. \"“Adversarial Examples” for Proof-of-Learning.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[2] Shavit, Yonadav. \"What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring.\" arXiv preprint arXiv:2303.11341 (2023).\n\n**Q1:** *Line 77, could there be a comparison between the formal formulation of Proof-of-Learning and Proof-of-Training-Data?*\n\n**Response:** We will include this comparison, see our response to W1.\n\n**Q2:** *Section 4, as most of the Proof-of-Learning experiments have been done on image datasets like CIFAR, I am curious if PoTD could perform similarly well on those image datasets.*\n\n**Response:** As mentioned in lines 312-316, we are also interested in future work testing PoTD on new modalities. Our efforts were particularly focused on scaling PoTDs to much larger models than the traditional PoL literature, to demonstrate their practicality. Specifically, works like [3] consider ResNets on CIFAR (ResNet-50 has 25M parameters and CIFAR-10 has 60K examples and is 163 MB), whereas our results are shown LLMs with large text corpora (125M to 1B parameters and OpenWebText has ~9B tokens and is ~17GB).\n\n[3] Jia, Hengrui, et al. \"Proof-of-learning: Definitions and practice.\" 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.\n"}, {"review_id": "nhl5yPh6An", "reviewer": "Reviewer_G7nA", "summary": "The authors propose Proof of Training Data (PoTD), a variant of Proof-of-Learning (PoL) protocols that focuses on training set attacks, rather than the training algorithm itself. A valid PoTD protocol should be able to, at least in theory, spot when a machine learning model has been trained on a different training set than the one declared by the learner. As in PoL, the learner is required to provide a full transcript of the training process, including training data, code and intermediate checkpoints. Unfortunately, the task of verifying a training transcript is as computationally intensive as re-training the model from scratch. For this reason, the authors propose several heuristic strategies for PoTD, which rely on the fact that stochastic gradient ascent tends to first memorize and then forget the data it observes in each batch.", "questions": "Would it be possible to bypass the memorization heuristic by changing both the model and the data? Theoretical work has shown that only a small portion of a large neural network is important for achieving good predictive performance (lottery tickets). Once a learner has trained a good model on dataset D*, it should be possible to add (a large number of) redundant neurons and retrain those on D. Would this be a valid attack on PoTD?\n\nThe Equation between lines 172 and 173 is not obvious at first sight, and should be clarified. Also, please number all equation in the paper.", "limitations": "I commend the authors for beign upfront about the limitations of their work. All my concerns have been listed above.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "From the methodology standpoint, I truly appreciate the memorization heuristics proposed in the papers. In particular, they seems to be able to defend against a large number of different threat models.", "weaknesses": "The PoTD protocol proposed by the authors have considerable overlap with the existing PoL proposals. I am inclined to see it as a variant of these existing efforts, rather than a novel, independent idea.\n\nFurthermore, the defenses proposed in the paper are heuristics and have been tested on large language models only. The authors mention that their techniques may work differently on other neural architectures, learning tasks and training procedures. Therefore, I believe more experimentation is needed to confirm their usefulness.\n\nAdditionally, the PoTD protocol requires the learner to disclose their complete learning process. Thus, there is no way to protect the intellectual property of the learner. The authors are honest about this limitation, and claim it will be addressed as future work. However, I feel this makes the paper weaker.\n\nThe data addition attack presented in lines 256-267 seems the most interesting scenario to me. It is unfortunate that the heuristic technique proposed by the authors cannot defend against it.\n\nThe writing style of the paper could be improved. I report here some specific examples.\n\nLine 6, \"and flag if the model specific harmful or beneficial data sources\" is not a syntactically-correct English sentence. \n\nLine 30, the authors start discussing how to solve proof-of-training-data\nbefore giving a precise enough definition that the reader can follow.\n\nLine 87, c2 is missing from the definition of V (compare with Line 82). Also, why is the probability taken over c1, when c1 does not appear in any of the terms?\n\nLine 172-173, please number all equations.\n\nSection 3.3 is very dense and many important details are left for future work.\n\nFootnotes 4-9 occupy almost a quarter of the page and contain important information. I would prefer having them merged with the main text.\n\nLine 225, \"on trained\" should be \"trained on\".\n\nSection 5 introduces new concepts, new attacks, new defenses, new notation. Since this happens so late in the paper, it ends up being a bit overwhelming. Why not explicitly organise the whole paper by threat model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Below, we respond specific points made in your review:\n\n**W1:** *The PoTD protocol proposed by the authors have considerable overlap with the existing PoL proposals. I am inclined to see it as a variant of these existing efforts, rather than a novel, independent idea.*\n\n**Response:** We emphasize that though our solution shares a common structure with techniques from the PoL literature, the PoTD problem is fundamentally different, and requires different tools. \nFirst, we clarify the difference between PoL and PoTD. **PoL only checks if the Prover is capable of spending the compute** to perform a large training run, but does not check whether the training transcript disclosed **actually corresponds** to the training run that yielded $W^*$. For example, a Prover can do an original training run, and then replace 10% of the original training data with synthetic data as in [1] (e.g. to hide that data from the Verifier), and the resulting modified transcript **would still be a valid PoL**. PoTD is more ambitious in that it **checks whether the exact reported data transcript would actually result in** $W^*$. We will better highlight this distinction in the camera ready.\n\nWe also emphasize that existing methods from the literature, while being close to achieving PoL, are far from achieving PoTD, and our new methods are important for closing this gap. For example, segment-wise retraining fails to stop replacement-with-synthetic-data attacks, which are a major problem for PoTD (but less fatal for PoL); we address this with our defense in Section 3.3. Prior methods are also too inefficient to practically catch attacks that target a small fraction of segments (e.g. addition, subtraction, or few-segment interpolation), as a PoL-style Verifier could not afford to retrain a large fraction of original training segments to spot a few spoofed ones. \n\n**W2:** *Furthermore, the defenses proposed in the paper are heuristics and have been tested on large language models only. The authors mention that their techniques may work differently on other neural architectures, learning tasks and training procedures. Therefore, I believe more experimentation is needed to confirm their usefulness.*\n\n**Response:** We emphasize that our experiments on Pythia models provide validation of our method’s performance “in the wild”, as they used different architectures and training procedures (and thus are as close to a “random alternative draw of hyperparameters” as we can practically get). We encourage future work on additional modalities and architectures, and have prioritized large transformers as they are the primary focus of current regulatory discussions.\n\n**W3:** *Additionally, the PoTD protocol requires the learner to disclose their complete learning process. Thus, there is no way to protect the intellectual property of the learner. The authors are honest about this limitation, and claim it will be addressed as future work. However, I feel this makes the paper weaker.*\n\n**Response:** This assumption of Verifier access to training data and model checkpoints at verification-time is the standard assumption used in all prior works in the Proof-of-Learning literature. In fact, in contrast to prior defenses (such as retraining), our memorization-defense and data-order-defense do not require knowledge of the training hyperparameters, and are thus essentially out-of-the-box verifiable using only black-box API access, preserving confidentiality. We will update the Discussion to better clarify this contribution. (Catching data addition attacks still does require weights and training-hyperparameters access, hence our comment on the need for future work.) \n\n**W4:** *The data addition attack presented in lines 256-267 seems the most interesting scenario to me. It is unfortunate that the heuristic technique proposed by the authors cannot defend against it.*\n\n**Response:** We wish to clarify that our method does allow us to catch at-scale data addition attacks, so long as they are a significant fraction of the training data in that segment, e.g. >5%. We strongly concur that better defenses against data addition attacks would be an excellent focus of future work. \n\n**W5-end:** *The writing style of the paper could be improved. I report here some specific examples.*\n\n**Response:** Thank you for pointing out these edits. We will incorporate them into the final draft.\n\n**Q1:** _Would it be possible to bypass the memorization heuristic by changing both the model and the data? [...]Once a learner has trained a good model on dataset $D^*$, it should be possible to add (a large number of) redundant neurons and retrain those on D. Would this be a valid attack on PoTD?_\n\n**Response:** Your idea is a good one, and we suspect there may be a viable attack in this direction that would require new defenses. In its current form, this attack would not be resistant to segment-wise retraining, because when retraining any specific segment, either the real neurons or the fake neurons (or both) would not be correctly reproduced. An additional trick a Prover could add to this attack could be adding so many neurons that they dominate the reproduction-error term, and the “true” neurons appear as “noise” smaller than $\\epsilon$. One defense would just be to require higher levels of reproducibility (e.g. small epsilon, or even perfect reproducibility if possible). We are also not sure that it is at all straightforward to get two parts of a neural network to memorize two different datasets, and never interfere with each others’ predictions; the tricks required to do so may introduce additional artifacts a Verifier could spot.\n\n**Q2:** *The Equation between lines 172 and 173 is not obvious at first sight, and should be clarified.*\n\n**Response:** We will add a step-by-step derivation in the appendix."}, {"review_id": "SQCpqM4ZDI", "reviewer": "Reviewer_pydU", "summary": "This paper proposes a protocol for verifying that a model trainer submits data and learned weights, and a verifier checks whether the weights are correctly learned from the submitted data.\nSuch a protocol is useful for trustworthy AI. The paper defines the problem of Proof-of-Trainind-Data (PoTD). It is inspired by the previous work on \"Proof-of-Learning,\" but the setting of PoTD requires more. The paper gives a formal definition of PoTD protocol and argues that the protocol needs to satisfy some conditions to achieve the guarantee in a practical setting.", "questions": "n/a", "limitations": "The paper mentions to limitations in the end of the paper.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "\nThe motivation of the paper is interesting. The proposed PoTD protocol seems reasonable. ", "weaknesses": "\n1. The paper does not explain the motivation why we need the PoTD protocol and its impact. The paper says that some attacks can be treated by existing Proof-of-Learning methods. It seems trivial since  PoTD poses stronger requirements, as written in the paper. The paper should discuss the problems of existing PoL methods and how the proposed protocol can solve them.\n\n2. The only attack that the existing method cannot deal with but the proposed method can is a data subtraction attack. How important to deal with such an attack is not explained enough.\nMoreover, experimental results show how the proposed method performs but do not compare with baseline methods. Therefore, experiments are not enough to\nshow the effectiveness of the proposed methods. \n\n3. The presentation of the paper is hard to follow. The paper seems to consist of fragments of texts relationships between them are unclear. \nFor example, There is a formal definition of PoTD protocol in section 2, but the definition is never mentioned in the subsequent sections. Therefore, it is hard to judge whether the definition is reasonable or not. \nThe memorization heuristic introduced in Section 3.2 seems overly complex. If we use $-L(d, W)$ instead of M(d, W) in (3), then the values of PBQ and FBQ would not change since\nthe first term of (2) does not depend on data d and has no effect when evaluating $\\Delta_M(d^\\prime, W) >  \\Delta_M(d, W)$.  \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Below, we respond specific points made in your review:\n\n**W1:** *The paper says that some attacks can be treated by existing Proof-of-Learning methods. It seems trivial since PoTD poses stronger requirements, as written in the paper. The paper should discuss the problems of existing PoL methods and how the proposed protocol can solve them.*\n\n**Response:** We are not sure which statements you are referring to. Most of the attacks raised in the paper cannot be affordably addressed using existing Proof-of-Learning methods, and even for those that can, our solutions are more efficient or have other benefits (e.g. only requiring inference and not retraining). We provide specific examples of attacks that PoL methods cannot address, but that our new methods can:\n- **PoL-style retraining is known to be potentially vulnerable to synthetic data attacks** [1], which allow the Prover to swap the real data out for carefully-crafted fake data. This is fatal for PoTD, and can also enable PoL spoofs (see [1]). Our new verified-data-shuffling defense in Section 3.3 stops this attack.\n- **Retraining is inefficient for catching small data lies**. Given 100 checkpoints, if a Prover wants to misreport 1 checkpoint’s worth of data, then to catch them w/ 80% probability the Verifier would have to rerun 80% of the original training run. However, with our method, the Verifier needs to spend as little as 1% (and only inference compute) to catch subtraction/interpolation attacks, and slightly more for addition attacks (which require retraining a couple segments to calibrate segment-magnitudes, but still yield a roughly order-of-magnitude improvement). Even if a Verifier is only worried about Provers misreporting 10 checkpoints, the Verifier would need to retrain $\\approx 16\\%$ of the checkpoints (see Appendix C of [2] for a similar derivation) to catch the lying Prover with 80% probability, much larger than the 1% or few percent required by our method.\n\nWe will include the above examples to better highlight the shortcomings of prior methods at achieving PoTD.\n\n[1] Zhang, Rui, et al. \"“Adversarial Examples” for Proof-of-Learning.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n[2] Shavit, Yonadav. \"What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring.\" arXiv preprint arXiv:2303.11341 (2023).\n\n**W2a:** *The only attack that the existing method cannot deal with but the proposed method can is a data subtraction attack. How important to deal with such an attack is not explained enough.*\n\n**Response:** To clarify, our methods not only enable detection of data subtraction attacks, but also block synthetic-data-substitution attacks (see Section 3.3, and [3] for the paper that originally provided the attack), and are the first to enable cost-effective detection of interpolation attacks and large-scale data addition attacks. Together, these block an attacker from far-underclaiming the training data they used, which is directly relevant for, e.g., policies around the amount of compute used for training. We will make this clearer in the paper.\n\n[3] Zhang, Rui, et al. \"“Adversarial Examples” for Proof-of-Learning.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n\n**W2b:** *Moreover, experimental results show how the proposed method performs but do not compare with baseline methods.*\n\n**Response:** Most of our defenses have no relevant baselines, such as data subtraction attacks on individual segments. In principle, one could run PoL-style retraining at a massive scale, but this would be far too expensive to be practically relevant.\n\n**W3a:** *There is a formal definition of PoTD protocol in section 2, but the definition is never mentioned in the subsequent sections.*\n\n**Response:** To clarify the paper’s structure, Section 2 provides a theoretical definition for PoTD, but proving such a theoretical guarantee is in practice currently impossible for large-scale NN training. Instead, just as in the original PoL paper, in Section 3.3 and 5 we propose a set of attacks, which are intended to approximate the full set of attacks on PoTD. We then show that our solution is robust to these attacks, and thereby demonstrate that our defenses heuristically satisfy Definition 1, for current attacks in the literature. We will clarify the set of considered attacks explicitly in Section 2, and enumerate all the attacks in a single location. \n\n**W3b:** *The memorization heuristic introduced in Section 3.2 seems overly complex. If we use $-L(d, W)$ instead of M(d, W) in (3), then the values of PBQ and FBQ would not change since the first term of (2) does not depend on data d and has no effect when evaluating $\\Delta_\\mathcal{M}(d’,W) > \\Delta_\\mathcal{M}(d, W)$.*\n\n**Response:** You are correct that the existence of the normalization term $E_{d' \\in D_v}[L(d', W)]$ in our memorization heuristic $\\mathcal{M}$ does not affect the values of PBQ and FBQ. This could represent a 2x efficiency gain, and we will mention it in the final manuscript. This normalization may still be important in other cases for preserving the meaning of the Memorization Delta $\\Delta_\\mathcal{M}$, which is used to produce memorization-charts. Without the normalization term, it becomes hard to compare $\\Delta_\\mathcal{M}$ across segments, especially in the earlier stages of training when the changes in loss between checkpoints vary quite a bit across segments. To illustrate, we created Memorization Delta plots with and without the normalization term to illustrate this point (see attached PDF).\n"}]

Author rebuttal:
We thank the reviewers for their useful feedback, and are glad that many of them enjoyed the paper. We have written detailed responses to each reviewer’s comments, and thank the reviewers for their recommendations.

**For Reviewer pydU** we attach the figure referred to in our response.

Meta-review: Reviewers found this paper exciting and unanimously recommended accept.
Decision: Accept (poster)