SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Initialization attacks: An attacker can choose a "random" initialization that places  $W_0$  in a convenient position, such as close to the target  $W^*$ . Even if the Verifier uses statistical checks to confirm that the initialization appears random, these are sufficiently loose that an adversary can still exploit the choice of initialization [ZLD+22].
- Synthetic data/data reordering attacks: Given the current weight vector  $W_{i}$ , an attacker can synthesize a batch of training datapoints such that the resulting gradient update moves in a direction of the attacker's choosing, such as towards  $W^{*}$ . This could be done through the addition of adversarial noise to existing data points [ZLD+22], generating a new dataset [TJSP22], or by carefully reordering existing data points in a "reordering attack" [SSK+21].
We propose methods for preventing both of these attacks by forcing the Prover to use a certified-random weight initialization, and a certified-random data ordering. The randomized data ordering guarantees that the adversarial Prover cannot construct synthetic datapoints that induce a particular gradient, because it does not know the corresponding weights  $W$  at the time of choosing the datapoints  $D$ . Given a fixed data ordering, we discuss in Appendix E why it may be super-polynomially hard to find a certified-random weight initialization that, when fully trained, results in a particular  $W^{*}$ .
The Verifier can produce this guaranteed-random initialization and data order by requiring the Prover to use a particular random seed  $s$ , constructed as a function of the dataset  $D$  itself. This produces the initialization  $W_{0} = G_{r}(s)\in \mathbb{X}^{n}$  and data ordering  $S = G_{p}(s)$  using a publicly known pseudorandom generators  $G_{r}$  and  $G_{p}$ . The Prover can also construct a verifiable validation subset  $D_{v}$  by holding
${}^{4}{G}_{r}$  is a cryptographically-secure pseudorandom  $d$  -length vector generator,with postprocessing defined in the hyperparameters  $M$  ,and  ${G}_{p}$  is a publicly-agreed pseudorandom  $n$  -length permutation generator.  ${G}_{p}$  can be modified to repeat data multiple times to train for multiple epochs, or according to a randomized curriculum.
In practice, the statistical test to verify that the certified ordering was used will only be able to distinguish whether each data point  $d_{i} \sim D$  was trained in the assigned segment  $S_{i}$  or not. Therefore, for this protocol to apply a checkpoint must be saved at least twice per epoch,  $k \leq n / 2$ .
out the last  $n_v$  data-points in the permutation  $S$  from training. The Prover constructs  $s$  as follows. Assume that the dataset  $D$  has some initial ordering. Let  $H$  be a publicly-known cryptographic hash function. We model  $H$  as a random oracle, so that when composed with  $G_r$  or  $G_p$ , the result is polynomial-time indistinguishable from a random oracle.<sup>6</sup> This means that if a Prover wants to find two different seeds  $s_1, s_2$  that result in similar initializations  $W_{0;1}, W_{0;2}$  or two similar permutations  $S_1, S_2$ , they can find these by no more efficient method than guessing-and-checking. For large  $d$  and  $n$ , finding two nontrivially-related random generations takes exponential time. We construct the dataset-dependent random seed  $s$  as
$$
s (D, s _ {\text {r a n d}}) = H \left(H \left(d _ {1}\right) \circ H \left(d _ {2}\right) \circ \dots \circ H \left(d _ {a}\right) \circ s _ {\text {r a n d}}\right), \tag {8}
$$
where  $\{d_1, \ldots, d_a\} = D$ ,  $\circ$  is the concatenation operator, and  $s_{rand}$  is a Prover-chosen 32-bit random number to allow the Prover to run multiple experiments with different seeds. A Verifier given access to  $D$  (or only even just the hashes of  $D$ ) can later rederive the above seed and, using the pseudorandom generators, check that it produces the reported  $W_0$  and  $S$ .
The important element of this scheme is that given an initial dataset  $D^{*}$  and resulting data order  $S$ , modifying even a single bit of a single data point in  $D^{*}$  to yield a second  $D$  will result in a completely different data order  $S'$  that appears random relative to  $S$ . Thus, if we can statistically check that a sequence of checkpoints  $\mathcal{W}$  matches a data order  $S^{*}$  and dataset  $D^{*}$  better than a random ordering, this implies that  $D^{*}$  is the only efficiently-discoverable dataset that, when truthfully trained<sup>8</sup>, would result in the checkpoints  $\mathcal{W}$  and final weights  $W^{*}$ . We provide this statistical test in Appendix C.
This same approach can be extended to the batch-online setting, where a Prover gets a sequence of datasets  $D_1^*, D_2^*, \ldots$  and trains on each before seeing the next. The Prover simply constructs a new seed  $s(D_i^*, s_{rand})$  for each dataset  $D_i^*$ , and continues training using the resulting data ordering. This works so long as each  $D_i^*$  is large enough for a particular data-ordering to not be brute-forceable.
4.4 Putting It All Together
In Appendix A we sketch a complete protocol for combining these defenses complementarily to detect all of the attacks discussed in Section 6. The overall computational cost for the Verifier is  $O(n)$  training data-point hashes,  $O(\alpha n)$  model inferences for computing losses, and  $O(|Q|n)$  gradient computations for retraining transcript segments (where  $|Q|$  depends on hyperparameters that can be adjusted according on the Verifier's compute budget). Importantly, the Verifier's cost grows no worse than linearly with the cost of the original training run. If we run our tests using an  $\alpha = 0.01$  fraction of the points in each segment as done in our experiments below, the verification cost of computing our new tests in Sections 4.2 and 4.3 totals just  $1.3\%$  of the original cost of training, assuming inference is  $3 \times$  cheaper than training.
Our main experiments are run on GPT-2  $\left[\mathrm{RWC}^{+}19\right]$  with 124M parameters and trained on the OpenWebText dataset [GCPT19]. We use a batch size of 491,520 tokens and train for 18,000 steps ( $\sim$ 8.8B tokens), which is just under 1 epoch of training, saving a checkpoint every 1000 steps. See Appendix F for additional details. The data addition attack experiments in Section 6 further use the Github component of the Pile dataset  $\left[\mathrm{GBB}^{+}20\right]$  as a proxy for a Prover including additional data that is different from reported data. In addition to training our own models, we also evaluate Pythia checkpoints  $\left[\mathrm{BSA}^{+}23\right]$  published by EleutherAI, as they publish the exact data order used to train their models. We chose the 70M, 410M, and 1B-sized Pythia models trained on the Pile dataset with dedduplication applied. All experiments were done using 4 NVIDIA A40 GPUs.
<sup>6</sup>Since the random oracle model is known to be unachievable in practice, we leave the task of finding a more appropriate cryptographic primitive as an interesting direction for future work.
To enable a Prover to only reveal the required subset of data to the Verifier, it may be best to construct  $s$  using a Merkle hash tree.
It is still possible to construct multiple data sets  $D_{1}, D_{2}$ , and train on both, interleaving batches. This is not a uniqueness attack, but a data addition attack, and will be addressed in Section 6.
6 Empirical Attacks and Defenses
Below, we show that our methods address the last three attacks (Checkpoint Glue-ing, Data Subtraction, and Data Addition). We omit the synthetic initialization and synthetic data attacks of  $\mathrm{[FJT^{+}22,ZLD^{+}22]}$  as we addressed those in Section 4.3. All plots are from experiments using GPT-2; we include additional experiments in Appendix H. We do not claim that the attacks studied here are exhaustive, but provide them as a starting point to motivate future work.
Checkpoint Glue-ing Attack A known attack against Proof-of-Learning, which also applies to PoTD, is to "glue" two training runs  $W^{A}$  and  $W^{B}$  together and report a combined sequence of checkpoints  $\mathcal{W} = (W_{0}^{A},\dots ,W_{i}^{A},W_{j}\gg 0,\dots ,W_{final}^{B})$ . The resulting model  $W_{final}^{B}$  can be trained on undisclosed data prior to segment  $j$ , with the Prover never reporting this data to the Verifier. As highlighted by [JYCC+21], the size of the glued segment  $\| W_j^B -W_i^A\| _2$  will generally appear as an outlier in weight-space. We demonstrate this phenomenon in Figure 4. Following [JYCC+21], a Verifier could then check such suspicious segments via retraining. We demonstrate a second verification option using inference instead of training: the Verifier can check whether the checkpoint  $W_{j}^{B}$  has memorized not only the most recent data  $\Pi_{i}$ , but also the preceding data segments  $\Pi_{i - 1},\Pi_{i - 2},\ldots$ . The absence of long-term memorization is visible in the memorization heatmap in Figure 4.
Figure 4: Exploring how the defenses handle a simulated gluing attack, where the transcript switches from one GPT-2 training run to a second run after the 9th checkpoint. (We assume the Prover uses a certified data order (Section 4.3) on the sections before and after gluing.) From left to right: The norm of the weight-changes jumps abruptly during the gluing, causing the Verifier to flag that checkpoint as suspicious.; the Verifier creates a memorization plot (shown here with  $100\%$  sampling rate for clarity), and discovers the gluing by spotting that memorization of past checkpoints cuts off abruptly at the suspicious segment.; The same long-term memorization cutoff effect is visible plotting average  $\mathcal{M}$  for each data segment across time.
To avoid the spike in weight-space shown in Figure 5 when jumping from  $W_{i}^{A}$  to  $W_{j}^{B}$ , the attacker can break up the large weight-space jump into smaller jumps by artificially constructing intermediate checkpoints  $aW_{j}^{B} + (1 - a)W_{i}^{A}$  for several values of  $a$ . However, these interpolated checkpoints fail our memorization tests, as they are artificial and not the result of actual training (Figure 5).
Data Subtraction Attack In a data subtraction attack, a Prover claims the model has been trained on more points than it truly has. Detecting data subtraction attacks could enable a Verifier to detect overclaiming by model providers, including claiming to have included safety-related data when they secretly did not. Subtraction can also be used to hide data addition attacks, as combining the two attacks would mean the segment was still trained on the correct number of datapoints, thus suppressing the weight-change-plot signature used to catch data addition (as in Figure 7). We demonstrate the effectiveness of an efficient memorization-based approach for detecting subtraction, described in Section 4.2. Leveraging the subtraction-upper-bound test from Equation 7, we see in Figure 6 that the upper-bound heuristic  $\lambda (\Pi ,p,W_i)$  is surprisingly tight, consistently differentiating between no-subtraction segments and even small subtraction attacks. Still, even if  $\lambda (\Pi_{i},p,\mathcal{W}) > z$  for some large  $z\gg 0$ , this is only an upper bound on the quantity of data subtraction, and does not prove that a  $z$ -fraction of points were subtracted. The Verifier can instead use this test as an indicator to flag segments for retraining, which would confirm a subtraction attack. (That retraining would
Figure 5: Simulating an interpolation attack by training a GPT-2 model until the 5th checkpoint, and then linearly-interpolating to a final checkpoint. On the left, we show that an attacker can carefully choose interpolation points to mask any irregularities in validation loss. (The green line perfectly overlaps with the blue line.) Nonetheless, on the right, we see a clear signature in the memorization plot, computed using only  $1\%$  of data: the typical memorization pattern along the diagonal does not exist for the interpolated checkpoints. For each row corresponding to a data segment  $\Pi_i$ , a box marks the maximal- $\mathcal{M}$  checkpoint. The box is red if the checkpoint is a match  $W_i$ , and magenta if there is no match and the test fails  $W_{j \neq i}$ .
Figure 6: On the left, we simulate a data subtraction attack with different levels of subtraction (0%, 5%, 50%, or 95%) in a GPT-2 training run. The plots show the results of computing the subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  for each checkpoint, using just  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.  $\lambda$  estimates the maximum level of data subtraction in each segment. We see that  $\lambda$  provides a surprisingly tight upper bound for the honestly-trained segment, while providing no such upper bound for the larger subtraction attacks. To illustrate the logic behind this test, on the right, we show how a  $50\%$  subtraction attack can create a bimodal distribution of  $\Delta_{\mathcal{M}}$  values.  $\lambda$  captures the relative weight of the left mode.
result in a different weight vector can be inferred from the plot of the  $50\%$  -addition attack in Figure 7). Appendix H.2 explores the test's performance on the suite of Pythia models.
Data Addition Attack Addition attacks occur when a Prover in addition to training on the declared dataset  $D$  and data sequence  $\Pi$ , trains on additional data  $D'$  without reporting it to the Verifier. This can be used to secretly enhance a model's capabilities, install backdoors, or train on restricted data. This attack cannot be detected using memorization analysis (Figure 7), because the Verifier does not know and cannot test points  $d' \in D'$ . However, Figure 7 also shows that significant amount of data addition, or data addition from a different distribution, causes a larger weight-displacement in that segment, which can be detected by looking at the segment's magnitude. If a Prover tries to hide this by deleting an equally-sized subset of points from  $D$ , that can be detected as a subtraction attack. We leave exploration of other attacks and defenses, like changing learning rates, to future work.
This raises the problem of how to estimate the "expected" segment length, which may require retraining on a chosen subset of segments, and interpolating to estimate other segments' lengths. If, to hide the non-smoothness of adding extra data to a single segment, the Prover adds data uniformly throughout a large fraction of the training run, then choosing even a small number of segments randomly should be sufficient to catch at least one offending segment with high probability. Unfortunately, these defenses would not detect an attacker that adds a modest amount of data within a small number of segments, that cannot be detected by the segment-magnitude or data subtraction defenses.
Figure 7: Simulating a data addition attack by picking a single segment (1st, 10th, or 18th), and adding  $50\%$  more data from the same distribution (OpenWebText), or  $5\%$  more data from a different distribution (Github), or no data addition (truthful reporting). Results are shown with error bars across 4 random seeds. From left to right: the memorization test with  $1\%$  of samples does not spot any differences even with  $50\%$  data addition; Plotting weight-changes between checkpoints, a Verifier can see a suspicious spike at the attacked segments; The Verifier retransmits the suspicious segments and checks the distance between the reported and re-executed checkpoint weights.
This work contributes to an emerging societal effort to develop practical and robust tools for accountability in the large-scale development of AI models. The statistical tests we introduce are best taken as an opening proposal. Future work could propose clever new attacks that break this protocol, or better yet, create new defenses that efficiently detect more, and subtler, attacks and enable trustworthy verification of ML models' training data.
Experimental Limitations This work provides suggestive evidence for the local-memorization phenomenon, but further study is needed across additional modalities, architectures, and training recipes in order to determine its broad applicability. Encouragingly, we find in Appendix G that local-memorization gets even stronger as models get larger, though memorization appears weaker near the end of training as the learning rate shrinks. The paper's experiments only include language models, in part because they are a current priority for audits. The memorization tests used may need to be adjusted for models trained with less data on many epochs, such as image models  $\left[\mathrm{YZS}^{+}22\right]$ .
Attacks Our Protocol Does Not Catch There are several remaining directions for attacks. The attacks explored above can be composed in new ways, and it may be possible for compositions of attacks to undermine the defenses that would otherwise detect each attack individually. The method also does not address small-scale data additions, and thus cannot yet detect copyright violations or spot inserted backdoors  $\left[\mathrm{XWL}^{+}21\right]$ . It also cannot detect attacks based on small-norm modifications to the weights, which could be used to insert backdoors  $\left[\mathrm{BISZ}^{+}22\right]$ . Finally, attacks could be masked with cleverly chosen hyperparameters, such as by using a temporary lower-than-reported learning rate to shrink large changes in  $W$ . Exploring whether such attacks are feasible without degrading learning performance – and identifying defenses – is an interesting direction for future work.
Applicability to Different Training Procedures We attempted to make our procedure as agnostic as possible to the details of the training procedure, and believe it will be compatible with most training procedures for large models in use today. However, our protocol does not apply to online or reinforcement learning, or to schemes that require multiple models to be co-trained  $\left[\mathrm{GPAM}^{+}20\right]$ , as the data is unknown in advance. This means the uniqueness defense cannot be applied (Section 4.3). Finding methods for defending against non-uniqueness attacks even in the online setting is a valuable direction for future work.
Maintaining Privacy and Confidentiality One significant challenge to using this protocol in practice is that, just like with PoL [JYCC+21], it requires that the Prover disclose confidential information to the Verifier, including training data, model weights, and code. In principle, the Prover may only need to disclose hashes of the data and weights to the Verifier, with the matching full data and weights only ever supplied on the secure cluster during verification.
We thank Nicolas Papernot, Anvith Thudi, Jacob Austin, Cynthia Dwork, Suhas Vijaykumar, Rachel Cummings Shavit, Shafi Goldwasser, Hailey Schoelkopf, Keiran Paster, Ariel Procaccia, and Edouard Harris for helpful discussions. DC was supported by NSERC CGS-D, and DC and YS are supported by Open Philanthropy AI Fellowships.
$\left[\mathrm{JYCC}^{+}21\right]$  Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP), pages 1039-1056. IEEE, 2021.
[KGG+22] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning. arXiv preprint arXiv:2202.09931, 2022.
[KRCC22] Zhifeng Kong, Amrita Roy Chowdhury, and Kamalika Chaudhuri. Forgeability and membership inference attacks. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security, pages 25-31, 2022.
$\left[\mathrm{OWJ}^{+}22\right]$  Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[RWC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. arXiv preprint arXiv:1901.11196, 2019.
[Sha23] Yonadav Shavit. What does it take to catch a Chinchilla? Verifying rules on large-scale neural network training via compute monitoring. arXiv preprint arXiv:2303.11341, 2023.
[SSK+21] Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson. Manipulating sgd with data ordering attacks. Advances in Neural Information Processing Systems, 34:18021-18032, 2021.
[TJSP22] Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007-4022, 2022.
[TSC+18] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018.
$\left[\mathrm{XWL}^{+}21\right]$  Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pages 103-120. IEEE, 2021.
[YZS+22] Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022.
[ZIL $^{+}$ 21] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938, 2021.
[ZLD+22] Rui Zhang, Jian Liu, Yuan Ding, Zhibo Wang, Qingbiao Wu, and Kui Ren. "adversarial examples" for proof-of-learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1408-1422, 2022.
A Combined Verification Protocol
We can unify the defenses of Section 4 into a combined defense protocol, which catches a wide swath of attacks, including all current attacks on from the Proof-of-Learning literature [FJT+22, ZLD+22].
A Prover gives the Verifier a transcript  $T = \{D, M, \mathcal{W}\}$  and a final weight vector  $W^*$ . The verifier proceeds to verify whether  $T$  is a valid training transcript through the following checks:
1. Check that  $\mathcal{W}$  ends in the claimed final weights  $W^{*}$ .
2. Given the dataset  $D$ , hash it to yield the seed  $s$  as in Section 4.3, and use that seed compute the resulting data order  $\Pi$  and validation subset  $D_v$ . (Alternatively, these hashes can be provided by the Prover, and only verified when each point is needed for the protocol.)
3. Check that  $W_0$  matches  $G_r(s)$ . If this fails, reject the transcript.
4. Create an empty list  $Q$  to store all suspicious-looking segments to retrain. For each segment  $W_{i}$ ,  $\Pi_{i}$ , include it in the list  $Q$  to retrain if it fails any of the following checks:
(a) Randomly select an  $\alpha$  fraction (e.g.,  $1\%$  of  $k$ ) of points  $\Pi_{i,\alpha}$  from  $\Pi_i$ . For each such point  $d \sim \Pi_{i,\alpha}$ , compute the losses on  $W_i$  and  $W_{i-1}$ , shorthanded as sets  $\mathcal{L}_{\Pi_i,i}$  and  $\mathcal{L}_{\Pi_i,i-1}$ . Similarly, pick an  $\alpha$  fraction of points from the validation set  $D_v$  and compute these points' losses on  $W_i$ , shorthanded as  $\mathcal{L}_{D_v,i}$ . (The validation loss on  $W_{i-1}$ ,  $\mathcal{L}_{D_v,i-1}$ , should've already been computed when looping on the previous segment.) Also, randomly select an  $\alpha k$  subset of data points  $D_t$  from across all training segments  $\Pi$ , and compute these points' losses on  $W_i$ ,  $\mathcal{L}_{D_t,i}$ .