SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dami Choi*
Yonadav Shavit*
yonadav@g.harvard.edu
David Duvenaud
duvenaud@cs.toronto.edu
It is important that consumers and regulators can verify the provenance of large neural models to evaluate their capabilities and risks. We introduce the concept of a "Proof-of-Training-Data": any protocol that allows a model trainer to convince a Verifier of the training data that produced a set of model weights. Such protocols could verify the amount and kind of data and compute used to train the model, including whether it was trained on specific harmful or beneficial data sources. We explore efficient verification strategies for Proof-of-Training-Data that are compatible with most current large-model training procedures. These include a method for the model-trainer to verifiably pre-commit to a random seed used in training, and a method that exploits models' tendency to temporarily overfit to training data in order to detect whether a given data-point was included in training. We show experimentally that our verification procedures can catch a wide variety of attacks, including all known attacks from the Proof-of-Learning literature.
How can we verify the capabilities of large machine learning models? Today, such claims are based on trust and reputation: customers and regulators believe that well-known companies building AI models wouldn't lie about the training data used in their models. However, as the ability to build new AI models proliferates, users need to trust an ever-larger array of model providers at their word, and regulators may increasingly face malicious AI developers who may lie to appear compliant with standards and regulations. Worse, countries developing militarily-significant AI systems may not trust each others' claims about these systems' capabilities, making it hard to coordinate on limits.
AI developers can enable greater trust by having a third party verify the developer's claims about their system, much as the iOS App Store checks apps for malicious code. Current black-box approaches to model auditing allow some probing of capabilities [Cen23], but these audits' utility is limited and a model's capabilities can be hidden [GTB22, GKVZ22]. An auditor can more effectively target their examination if they also know the model's training data, including the total quantity, inclusion of data likely to enable specific harmful capabilities (such as texts on cyber-exploit generation), and inclusion of safety-enhancing data (such as instruction-tuning [ \mathrm{OWJ}^{+}22 ]). However, if auditors rely on AI developers to self-report the data, developers could falsify their reports. This uncertainty limits the trust such audits can create.
In this work, we define the problem of Proof-of-Training-Data (PoTD): a protocol by which a third-party auditor (the "Verifier") can verify which data was used by the developer (the "Prover") to train a model. Our verification procedures assume that the Verifier can be given access to sensitive information and IP (e.g., training data, model weights) and is trusted to keep it secure; we leave the additional challenge of simultaneously preserving the confidentiality of the training data and model weights to future work. In principle, one could solve PoTD by cryptographically attesting to the
results of training on a dataset using delegated computation [CKV10]. However, in practice such delegation methods are impractically slow, forcing us to turn to heuristic verification approaches.
Inspired by the related literature on "Proof-of-Learning" (PoL)[JYCC+21], we propose that model-trainers disclose a training transcript to the Verifier, including training data, training code, and intermediate checkpoints. In Section 4, we provide several verification strategies for a Verifier to confirm a training transcript's authenticity, including new methods that address all published attacks in the Proof-of-Learning literature. We demonstrate the practical effectiveness of our defenses via experiments on two language models (Section 6). Our methods can be run cheaply, adding as little as  $1.3\%$  of the original training run's compute. Further, we require no change to the training pipeline other than fixing the data ordering and initialization seeds, and storing the training process seeds for reproducibility. Still, like PoL, they sometimes require re-running a small fraction of training steps to produce strong guarantees.
The verification strategies we describe are not provably robust, but are intended as an opening proposal which we hope motivates further work in the ML security community to investigate new attacks and defenses that eventually build public confidence in the training data used to build advanced machine learning models.
We build on [Sha23], which sketches a larger framework for verifying rules on large-scale ML training. It defines, but does not solve, the "Proof-of-Training-Transcript" problem, a similar problem to Proof-of-Training-Data that additionally requires verifying hyperparameters.
Proof-of-Learning.  $\left[\mathrm{JYCC}^{+21}\right]$  introduce the problem of Proof-of-Learning (PoL), in which a Verifier checks a Prover's ownership/copyright claim over a set of model weights by requiring the Prover to prove that they did at least as much computational work as was required to have trained the original model. The Prover's proof consists of reporting a sequence of valid weight checkpoints that led to the final weights. The Verifier then re-executes training between a few checkpoints and checks that the results are similar to confirm (or reject) the report's correctness. Unlike PoL, Proof-of-Training-Data requires proving what exact data was used to produce a model. This is strictly harder: any valid PoTD protocol can serve as a solution to PoL, as compute cost can be derived from the size of the dataset. Furthermore, PoL only requires robustness to adversaries that use less computation than the original training run, whereas PoTD targets all computationally-feasible adversaries.
Several works have successfully attacked the original PoL scheme; our proposed defenses defeat all published attacks.  $\mathrm{[ZLD^{+}22]}$  and [KRCC22] provide methods for forging fake data that will (during retraining) interpolate between chosen checkpoints, thus allowing the production of fake transcripts. (See Section 4.3 for our mitigation.)  $\mathrm{[FJT^{+}22]}$  exploit the fact that, due to the prohibitive cost of retraining, the Verifier can only retrain a tiny fraction of checkpoints. By falsifying only a modest fraction of checkpoints, a Prover can prevent their fake transcript being caught with high probability. (See Section 4.2 for our mitigation, which introduces a method for verifying checkpoints that is much cheaper than retraining, and can thus be applied to all checkpoints.)
Memorization during training.  $\left[\mathrm{ZIL}^{+21}\right]$  introduce the notion of counterfactual memorization (the average difference in model performance with and without including a specific point in training) that is most similar to our own, and use it to investigate different training points' effects on final model performance. [FZ20] examine which datapoints are most strongly memorized during training by using influence functions, but they focus on the degree of memorization only at the end of training.  $\left[\mathrm{BPS}^{+23}\right]$  show that per-datapoint memorization of text (as measured by top-1 recall) can be somewhat reliably predicted based on the degree of memorization earlier in training.  $\left[\mathrm{KGG}^{+22}\right]$  analyze pointwise loss trajectories throughout training, but do not focus specifically on the phenomenon of overfitting to points in the training set.
3 Formal Problem Definition
In the Proof-of-Training-Data problem, a Prover trains an ML model and wants to prove to a Verifier that the resulting target model weights  $W^{*}$  are the result of training on data  $D^{*}$ . If a malicious Prover used training data that is against the Verifier's rules (e.g., terms of service, regulatory rules) then that
Prover would prefer to hide  $D^{*}$  from the Verifier. To appear compliant, the Prover will instead lie and claim to the Verifier that they have used some alternative dataset  $D \neq D^{*}$ . However, the Prover will only risk this lie if they believe that with high probability they will not get caught (making them a "covert adversary" [AL07]). The goal of a Proof-of-Training-Data protocol is to provide a series of Verifier tests that the Prover would pass with high probability if and only if they truthfully reported the true dataset that was used to yield the model  $W^{*}$ .
Let  $D \in \mathbb{X}^n$  be an ordered training dataset. Let  $M$  contain all the hyperparameters needed to reproduce the training process, including the choice of model, optimizer, loss function, random seeds, and possibly details of the software/hardware configuration to maximize reproducibility.
Definition 1. A valid Proof-of-Training-Data protocol consists of a Prover protocol  $\mathcal{P}$ , Verifier protocol  $\mathcal{V}$ , and witnessing template  $\mathbb{J}$  that achieves the following. Given a dataset  $D^{*}$  and hyperparameters  $M^{*}$ , an honest Prover uses  $\mathcal{P}$  to execute a training run and get  $(W^{*}, J^{*}) = \mathcal{P}(D^{*}, M^{*}, c_{1})$ , where  $W^{*} \in \mathbb{R}^{d}$  is a final weight vector,  $J^{*} \in \mathbb{J}$  is a witness to the computation, and  $c_{1} \sim C_{1}$  is an irreducible source of noise. The Verifier must accept this true witness and resulting set of model weights with high probability:  $\mathrm{Pr}_{c_{1} \sim C_{1}, c_{2} \sim C_{2}}[\mathcal{V}(D^{*}, M^{*}, J^{*}, W^{*}, c_{2}) = 1] \geq 1 - \delta_{1}$ , where  $\delta_{1} \ll 1/2$  and  $c_{2}$  is the randomness used by the Verifier.
Conversely,  $\forall$  computationally-feasible probabilistic adversaries  $\mathcal{A}$  which produce proofs  $(D,M,J) = \mathcal{A}(D^{*},M^{*},J^{*},W^{*},c_{3})$  where  $D\neq D^{*}$  and  $c_{3}\sim C_{3}$  is the randomness used by the adversary, the Verifier must reject all such proofs with high probability:  $\operatorname{Pr}_{c_1\sim C_1,c_2\sim C_2,c_3\sim C_3}[\mathcal{V}(D,M,J,W^*,c_2) = 0]\geq 1 - \delta_2$  where  $\delta_2\ll 1 / 2$
Following the literature on the related Proof-of-Learning problem  $\mathrm{[JYCC^{+}21]}$ , we use as a witness the series of  $m$  model weight checkpoints  $J^{*} = \mathcal{W} = (W_{0},W_{1},\dots ,W_{m - 1},W^{*})$ . Model weight checkpoints are already routinely saved throughout large training runs; we assume a checkpoint is saved after training on each  $k = n / m$ -datapoint segment. During verification, the Prover provides the Verifier with the training transcript  $T = \{D,M,\mathcal{W}\}$ , which the Verifier will then test to check its truthfulness.
In practice, we cannot yet pursue provable robustness to all probabilistic adversaries  $\mathcal{A}$ , due to the nascent state of the theoretical foundations of deep learning [FJT+22]. Instead, as is done in the PoL literature, we approximate robustness to  $\mathcal{A}$  by proposing a range of adversaries and then showing a protocol that defeats them. In particular, we will consider two major types of attacks (and their combinations):
- Non-Uniqueness Attack: The Prover forges a different  $D \neq D^{*}$  and  $M$  (and corresponding checkpoints  $\mathcal{W}$ ) that would, if training was reexecuted, also lead to  $W^{*}$ . In general, it is easy to produce such transcripts (e.g., initialize training at  $W_{0} = W^{*}$  and take no steps). In Section 4.3 we propose constraints on  $\mathcal{P}$  that block such attacks.
- Data Subtraction Attack: The Prover reports training on  $D$ , but secretly only trains on a subset  $D^{*} \subsetneq D$ .
- Data Addition Attack: The Prover reports training on  $D$ , but secretly trains on an additional set of data  $D^{*} \supseteq D$ .
- Checkpoint Glue-ing Attack: The Prover reports a pair of successive checkpoints  $(W_{i}, W_{i+1}) \subset \mathcal{W}$ , but  $W_{i+1}$  was not produced from  $W_{i}$  by a training procedure at all.
We address the latter three attacks in Section 6.
As a brute-force solution to Proof-of-Training-Data, the Verifier could simply re-execute the complete training process defined by  $T$ , and check that the result matches  $W^{*}$ . However, beyond technical complications $^{2}$ , doing so is far too computationally expensive to be done often; a government Verifier would need to be spending as much on compute for audits as every AI developer combined. Therefore any verification protocol  $\mathcal{V}$  must also be efficient, costing much less than the original training run. Inevitably, such efficiency makes it near certain that the Verifier will fail to catch spofs  $D \neq D^{*}$  if  $D$  only differs in a few data points; in practice, we prioritize catching spofs which deviate on a
2This would also fail in practice because of irreducible hardware-level noise which means that no two training runs return exactly the same final weight vector  $\mathrm{[JYCC^{+}21]}$ . a transcript could still be examined piecewise, as done in  $\mathrm{[JYCC^{+}21]}$ ; for more, see Section 4.1.
substantial fraction of points in  $D^{*}$ . Though we do not restrict to a particular definition of dataset deviations, we list several possibilities relevant for different Verifier objectives in Appendix D.
4 Verification Strategies
We provide several complementary tools for detecting whether a transcript  $T$  is spoofed. Combined, these methods address many different types of attacks, including all current attacks from the PoL literature [ZLD+22, FJT+22].
4.1 Existing Tools from Proof-of-Learning
Our protocol will include several existing spoof-detection tools from the Proof-of-Learning literature [JYCC+21], such as looking for outliers in the trajectory of validation loss throughout training, and plotting the segment-wise weight-change  $\| W_{i} - W_{i-1}\|_{2}$  between the checkpoints  $\mathcal{W}$ . The most important of these existing tools is the segment-wise retraining protocol of [FJT+22]. Let  $R(W_{i-1},\Pi_i,c;M)$  be the model training operator that takes in a weight checkpoint  $W_{i-1}$ , updates it with a series of gradient steps based on training data sequence  $\Pi_i$  (describing the order in which the Prover claims data points were used in training between checkpoints  $W_{i-1}$  and  $W_{i}$ , which may be different from the order of the dataset  $D^*$ ), hyperparameters  $M$ , and hardware-noise-randomness  $c\sim C$ , and then outputs the resulting weight checkpoint  $W_{i}$ . Transcript segment  $i$  is  $(\epsilon,\delta)$ -reproducible if for the pair of checkpoints  $(W_{i-1},W_{i})$  in  $\mathcal{W}$ , the reproduction error (normalized by the overall segment displacement) is small:
$$
\Pr_ {c \sim C} \left(\frac {\| \hat {W} _ {i} - W _ {i} \| _ {2}}{\frac {\| \hat {W} _ {i} - W _ {i - 1} \| _ {2} + \| W _ {i} - W _ {i - 1} \| _ {2}}{2}} <   \epsilon\right) > 1 - \delta \quad \text {w h e r e} \quad \hat {W} _ {i} = R \left(W _ {i - 1}, \Pi_ {i}, c; M\right). \tag {1}
$$
The values  $\epsilon$  and  $\delta$  trade off false-positive vs. false-negative rates; see [JYCC $^{+}$ 21, FJT $^{+}$ 22] for discussion. The Verifier can use this retraining procedure as a ground-truth for verifying the faithfulness of a suspicious training segment. However, this test is computationally-intensive, and can thus only be done for a small subset of training segments. Our other verification strategies described in Sections 4.2 and 4.3 will be efficient enough to be executable on every training segment.
4.2 Memorization-Based Tests
The simplest way for a Prover to construct a spoofed transcript ending in  $W^{*}$  is to simply make up checkpoints rather than training on  $D^{*}$ , and hope that the Verifier lacks the budget to retrain a sufficient number of checkpoints to catch these spoofed checkpoints. To address this, we demonstrate a heuristic for catching spoofed checkpoints using a small amount of data, based on what is to the best of our knowledge a previously-undocumented phenomenon about local training data memorization.
Machine learning methods notoriously overfit to their training data  $D$ , relative to their validation data  $D_v$ . We can quantify the degree of overfitting to a single data point  $d$  on a loss metric  $\mathcal{L}:\mathbb{X}\times \mathbb{R}^{|W|}\to \mathbb{R}$  relative to a validation set  $D_v$  via a simple memorization heuristic  $\mathcal{M}$ :
$$
\mathcal {M} (d, W) = \mathbb {E} _ {d ^ {\prime} \in D _ {v}} [ \mathcal {L} (d ^ {\prime}, W) ] - \mathcal {L} (d, W). \tag {2}
$$
Recall that  $\Pi_{i}$  is the sequence of data points corresponding to the  $i$ th segment of the training run. One would expect that in checkpoints before data segment  $i$ , for data points  $d \in \Pi_{i}$ , memorization  $\mathcal{M}(d, W_{j < i})$  would in expectation be similar to the validation-set memorization; after data-segment  $i$ , one would expect to see higher degrees of overfitting and therefore  $\mathcal{M}(d, W_{j \geq i})$  would be substantially higher. We find evidence for this effect in experiments on GPT-2-Small [RWC+19] and the Pythia suite [BSA+23]). As shown in Figures 1 and 2, when a Prover reports the true training data, on average the greatest memorization occurs where  $\Pi_{i}$  and  $W_{j = i}$  match. We corroborate this finding with additional experiments on a range of models in Appendix G. The finding is even clearer if we look at jumps in memorization level, which we call the Memorization Delta  $\Delta_{\mathcal{M}}$ :
$$
\Delta_ {\mathcal {M}} (d, i; \mathcal {W}, D _ {v}, \mathcal {L}) = \mathcal {M} (d, W _ {i}) - \mathcal {M} (d, W _ {i - 1}). \tag {3}
$$
To test whether each reported checkpoint  $W_{i}$  resulted from training on at least some of the segment training data  $\Pi_{i}$ , a Verifier can compute a memorization plot like the one shown in Figure 1. Such
Figure 1: Plots from a GPT-2 experiment, demonstrating the local memorization effect. The maximum score for each data segment (row) is marked with a red box. The largest average memorization for data sequence  $\Pi_{i}$  occurs at the immediately-subsequent checkpoint  $W_{i}$ . From left to right: plots of the average loss  $\mathcal{L}$ ; memorization  $\mathcal{M}$ ; and memorization-delta  $\Delta_{\mathcal{M}}$ ; along with the average memorization over time for each segment  $\Pi_{i}$ , recentered such that  $W_{i}$  is at  $x = 0$ .
Figure 2: Plots of the memorization  $\mathcal{M}$  on other types of training runs, similar to Figure 1. For efficiency, plots of the Pythia models use only  $10\%$  of training data, and look only at a window of checkpoints around  $W_{i}$ . From left to right:  $\mathcal{M}$  for checkpoints near the middle of the Pythia (70M) training run shows the same pattern as GPT-2; checkpoints near the middle of the Pythia (1B) training run show that the phenomenon gets clearer as the model size increases;  $\mathcal{M}$  for a GPT-2 run with two epochs over the same data (with random data order each epoch; the first epoch ends at checkpoint 9) to demonstrate that the effect is present over multiple epochs;  $\mathcal{M}$  for a GPT-2 run using a random data order other than  $\Pi$  shows that the effect is tied to the training data sequence itself.
plots can be computed more efficiently by sampling only a small fraction  $\alpha$  of the training data  $\Pi$ , and by plotting only a few checkpoints  $W_{i - \beta},\ldots ,W_{i + \beta}$  for each segment  $\Pi_i$ .
We can further harness this memorization phenomenon to test whether on segment  $i$ , rather than training on the full claimed data sequence  $\Pi_{i}$  and yielding  $W_{i}$ , the Prover secretly skipped training on at least a  $\kappa$ -fraction of the points in  $\Pi_{i}$  and yielded  $W_{i}^{\prime}$ . Consider the odds that, for  $d \sim \Pi_{i}$ ,  $\Delta_{\mathcal{M}}(d, W_{i})$  happens to fall in the bottom  $p$ -probability quantile of the validation set  $D_{v}$ 's  $\Delta_{\mathcal{M}}$  values on  $W_{i}$ :
$$
\mathrm {P B Q} (d, p, W _ {i}) = \mathbb {I} \left(\mathbb {E} _ {d ^ {\prime} \sim D _ {v}} \mathbb {I} \left(\Delta_ {\mathcal {M}} \left(d ^ {\prime}, W _ {i}\right) > \Delta_ {\mathcal {M}} \left(d, W _ {i}\right)\right) \leq p\right) \tag {4}
$$
$$
\operatorname {F B Q} (\Pi , p, W _ {i}) = \mathbb {E} _ {d \sim \Pi} \left[ \operatorname {P B Q} (d, p, W _ {i}) \right], \tag {5}
$$
where  $\mathbb{I}$  is the indicator function, PBQ stands for "Point is Below Quantile", and FBQ stands for "Fraction Below Quantile". We can see in Figure 3 that, as expected, when the points in  $\Pi_{i}$  are all included in training,  $\mathrm{FBQ}(\Pi_{i},p,W_{i})$  is small compared to  $\mathrm{FBQ}(\Pi_{j\neq i},p,W_i)$ . If many points were secretly excluded, as in  $W_{i}^{\prime}$ , we should expect that  $\mathrm{FBQ}(\Pi_i,p,W_i^{\prime})$  should be higher and closer to  $\mathrm{FBQ}(D_v,p,W_i^{\prime})$ , where  $D_{v}$  is the validation set. If the Prover secretly excluded a greater than  $\kappa$ -fraction of data points in  $\Pi_{i}$  thus yielding  $W_{i}^{\prime}$ , then we should expect that:
$$
\operatorname {F B Q} \left(\Pi_ {i}, p, W _ {i} ^ {\prime}\right) = (1 - \kappa) \operatorname {F B Q} \left(\Pi_ {i}, p, W _ {i}\right) + \kappa \operatorname {F B Q} \left(D _ {v}, p, W _ {i} ^ {\prime}\right) \geq \kappa \cdot \operatorname {F B Q} \left(D _ {v}, p, W _ {i} ^ {\prime}\right) \tag {6}
$$
(see derivation in Appendix B). Rearranging terms, we get
$$
\lambda \left(\Pi_ {i}, p, W _ {i} ^ {\prime}\right) := \frac {\operatorname {F B Q} \left(\Pi_ {i} , p , W _ {i} ^ {\prime}\right)}{\operatorname {F B Q} \left(D _ {v} , p , W _ {i} ^ {\prime}\right)} \geq \kappa . \tag {7}
$$
$\lambda (\Pi_i,p,W_i^{\prime})$  can be estimated using a small fraction of training and validation datapoints, and can serve as an upper-bound estimate on  $\kappa$ , the fraction of  $\Pi_{i}$  secretly excluded from training  $W_{i}^{\prime}$ .<sup>3</sup> In
3The percentile-threshold  $p$  is left unspecified, but should be kept  $\ll 0.5$ . The test can be strengthened by varying the chosen fraction  $p$  and rerunning the analysis to confirm its insensitivity.
Figure 3: Exploring the pointwise memorization effect on GPT-2. From left to right: For each  $W_{i}$  and  $\Pi_{i}$ , we plot the fraction of points with  $\Delta_{\mathcal{M}}$  above the median  $1 - \mathrm{FBQ}(\Pi_i,0.5,W_i)$ , and see that in the diagonal segments, most individual points are above the median. This shows that memorization occurs pointwise, suggesting that it can be detected via sparse random sampling. The highest segment in each row is surrounded by a red box; Plotting the fraction of samples below the  $10\%$ ile, we see the fraction is uniquely low on diagonal tiles  $(\Pi_i,W_{j = i})$ , as predicted; two histograms comparing  $\Delta_{\mathcal{M}}$  for diagonal vs. nondiagonal weight checkpoints across all data segments, shows how the distributions may or may not overlap. Even at Checkpoint 1, the leftmost plot shows that  $\Pi_1$  has a (marginally) larger fraction of points above the median than any other data segment.
Section 6 we show that this heuristic can detect even small data subtractions in practice, and in Appendix H.2 we show the test's effectiveness across a range of percentiles  $p$  and segment-lengths  $k$ .
We also observe that  $\mathcal{M}$  gradually decreases over time from an initial peak immediately after the point's training segment. This echoes the many findings on "forgetting" in deep learning [TSC+18]. We show in Section 6 how this can be used to catch gluing attacks.
4.3 Fixing the Initialization and Data Order
As mentioned in Section 3, a Proof-of-Training-Data protocol needs to defend against non-uniqueness attacks, by making it difficult for a malicious Prover to produce a second transcript with  $D \neq D^{*}$  that, if training was legitimately executed, would also end in  $W^{*}$ . There are two well-known types of attacks the Prover might use to efficiently produce such proofs: