SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- If the Verifier wants to plot complete memorization plots, for example as a sanity check or to use in checking for a Glue-ing attack as described in Section 6, they can also compute the losses on  $2\beta$  nearby weight checkpoints  $W_{i - \beta},\ldots ,W_{i + \beta -1}$ . However, this is not part of the core protocol, and will not be counted in its sample complexity.
(b) Compare the values in  $\mathcal{L}_{\Pi_i,i}$  and  $\mathcal{L}_{D_t,i}$  using the one-sided binomial hypothesis test described in Appendix C, to check that the model used the correct data ordering. If the test cannot reject the null hypothesis that  $\Pi_i$  does not match  $W_{i}$  better than randomly-selected training points  $D_{t}$ , then run the test on an even larger fraction of data points from  $\Pi_{i}$  and  $D$  to double-check the statistical significance of the null. If this still fails, it suggests that the transcript may have been spoofed. Investigate further (such as by reexecuting training on this segment) and/or reject the transcript.
(c) Using  $\mathcal{L}_{\Pi_i,i},\mathcal{L}_{\Pi_i,i - 1},\mathcal{L}_{D_v,i}$ , and  $\mathcal{L}_{D_v,i - 1}$ , calculate a data subtraction upper bound heuristic as in Section 4.2. If this heuristic fails to bound the data subtraction to below an acceptable small limit (which depends on how strongly the Verifier wants to catch even small subtraction attacks, and on the Verifier's retraining budget), add this segment to  $Q$ .
(d) Using  $\{\mathcal{L}_{D_{v,j}}\mid j\in 0,\dots ,m\}$ , compute the mean validation set loss across time, and check that it is smooth at  $W_{i}$  and doesn't increase or decrease discontinuously. If it does, add this segment to  $Q$ . (The tolerance should depend on the retraining budget.)
(e) Compute the distance  $\| W_{i} - W_{i - 1}\|$  from the previous checkpoint. Check that this delta is similar to nearby segments' weight-space deltas. If it is not, add this segment to  $Q$ . (The tolerance should depend on the retraining budget. We leave the question of how best to set this threshold, and of  $\sigma$  below, to minimize false negatives while avoiding reducing false positives that increase retraining costs, to future work.)
5. Randomly sample  $\sigma$  additional data segments from throughout training, and add them to  $Q$ . These additional segments are important to establish baseline estimates of segments' weight-space deltas across training, to ensure that there were no systematic data addition attacks at every segment. (Illegal data additions in every segment would shift the entire weight-change delta magnitude plot, thus suppressing anomalies in any one segment).
6. For each segment in the list  $Q$ , execute retraining and verify that the resulting weights  $\hat{W}_i$  are within an  $\epsilon$ -ball of the original reported weights  $W_i$  reported in the transcript.
To reduce noise when comparing validation performance across checkpoints, this  $\alpha$  subset of  $D_v$  should be the same across all evaluated checkpoints.
If any values in the re-trained weights fail to come within the tolerance  $\epsilon$ , that is significant evidence that the transcript has been spoofed, and warrants further investigation. For example, the segment can be retrained more times, to confirm that the weight-space variance across retraining results  $\hat{W}^{(1)}, \hat{W}^{(2)}, \ldots$  is sufficiently smaller than  $\epsilon$  such that the reported  $W_{i}$  is a clear outlier.
If all these tests pass, accept the transcript.
A.1 Complexity
The time costs of training, borne by the Prover are:
1.  $h \times |D|$ , where  $h$  is the cost of a hash, for generating the initial random seed.
2.  $s \times n$ , where  $s$  is the cost of a single gradient computation, and  $n$  is the number of training data points.
In comparison, the time costs to the Verifier (assuming the transcript is accepted) are:
1.  $h \times |D|$  hashes for verifying the initial weights.
2.  $(2 + 1 + 1) \times \alpha \times \frac{s}{3} \times n$  operations for computing the loss of an  $\alpha$  fraction of datapoints in  $\Pi_i$  on  $W_i$  and  $W_{i-1}$ , and another  $2\alpha$  fraction of points in  $D_t$  and  $D_v$ . We also assume that computing the loss requires  $\frac{1}{3}$  the number of operations as computing a gradient update, which is the standard ratio of inference vs. training when using backpropagation.
3.  $s \times n \times |Q| / m$  operations for retraining, where  $m$  is the total number of checkpoints in the training run.
B Subtraction Upper-Bound Heuristic Derivation
Let  $W_{i}$  be the checkpoint obtained by training on the full claimed data sequence  $\Pi_{i}$ , and let  $W_{i}^{\prime}$  be the checkpoint obtained by excluding  $\kappa$ -fraction of the points in  $\Pi_{i}$ . Our goal is to estimate  $\mathrm{FBQ}(\Pi_{i}, p, W_{i}^{\prime})$ , the fraction of points in  $\Pi_{i}$  that fall in the bottom  $p$ -probability quantile of  $D_{v}$ 's  $\Delta_{\mathcal{M}}$  values on  $W_{i}^{\prime}$ .
Let's start with the distribution of  $\Delta_{\mathcal{M}}(d,W_i)$ , where  $d\sim \Pi_{i}$ . We make the following two assumptions:
1. The subtracted  $\kappa$ -fraction of points are sampled randomly from  $\Pi_i$ .
2. The  $p$ -th percentile of the  $\Delta_{\mathcal{M}}$  values of  $D_v$ 's  $\Delta_{\mathcal{M}}$  values on  $W_i$  is the same as on  $W_i'$ .
Firstly, since  $\kappa$ -fraction of points were subtracted, we know that those subtracted points are distributed like the validation set (assumption 1). Therefore, the fraction of points in  $\Pi_i$  that lie in our quantile of interest that were subtracted is  $\kappa \mathrm{FBQ}(D_v,p,W_i')$ .
Next, we must consider the points that were not subtracted but lie in the quantile. Using assumption 2, we know that  $(1 - \kappa)$  fraction of points that were below the quantile remain in the quantile. Therefore, the fraction of points in  $\Pi_{i}$  that lie in our quantile of interest that were not subtracted is  $(1 - \kappa)\mathrm{FBQ}(\Pi_i,p,W_i)$ .
Putting it all together, we have
$$
\operatorname {F B Q} \left(\Pi_ {i}, p, W _ {i} ^ {\prime}\right) = \kappa \operatorname {F B Q} \left(D _ {v}, p, W _ {i} ^ {\prime}\right) + (1 - \kappa) \operatorname {F B Q} \left(\Pi_ {i}, p, W _ {i}\right) \tag {9}
$$
There are several things to note about this derivation. First of all, regardless of whether assumption 2 is correct, the second term of Equation 9 is  $\geq 0$  and therefore,  $\lambda$  is still an upper-bound. In practice, the actual fraction of subtracted points that lie in the quantile might not be exactly equal to  $\mathrm{FBQ}(D_v,p,W_i') = p$ . Therefore, if  $(1 - \kappa)\mathrm{FBQ}(\Pi_i,p,W_i)$  is very small and close to zero, the resulting upper-bound can be smaller than  $\kappa$ , which is the case in Figure 6.
Lastly, when computing FBQ values, we can use the unnormalized  $\mathcal{M}$  values to compute  $\Delta_{\mathcal{M}}$  as such:
$$
\Delta_ {\mathcal {M}} (d, i; \mathcal {W}, \mathcal {L}) = \mathcal {L} (d, W _ {i}) - \mathcal {L} (d, W _ {i - 1}). \tag {10}
$$
This does not affect the FBQ values, since the normalization term  $\mathbb{E}_{d' \in D_v}[\mathcal{L}(d', W)]$  does not depend on  $d$  and therefore and has no effect when evaluating  $\Delta_{\mathcal{M}}(d', W_i) > \Delta_{\mathcal{M}}(d, W_i)$ . Removing the normalization term could represent a 2x efficiency gain. We kept the normalization term in the main paper because it may still be important in other cases for preserving the meaning of the Memorization Delta  $\Delta_{\mathcal{M}}$ , which is used to produce memorization-charts. Without the normalization term, it becomes hard to compare  $\Delta_{\mathcal{M}}$  across segments, especially in the earlier stages of training when the changes in loss between checkpoints vary quite a bit across segments.
C Data Order Statistical Test
We want a statistical test that will tell the Verifier whether, for a given training dataset  $D$  and data ordering  $S$ , which together yield a data sequence  $\Pi$ , and for a given weight checkpoint  $W_{i}$ , the data segment sequence  $\Pi_{i} \in \mathcal{X}^{k}$  explains the memorization pattern of  $W_{i}$  better than a random data order/sequence  $\Pi_{i}'$  (which we assume is drawn randomly from  $D$ ). In particular, based on results from Section 4.2, we know that datapoints from the most recent training segment  $d \in \Pi_{i}$  tend to have higher memorization delta values  $\Delta_{\mathcal{M}}$  than the average point  $d' \in D$  from the overall training distribution  $D$ . Conversely, points from  $\Pi_{i}'$  would have no reliably greater  $\Delta_{\mathcal{M}}$  than the rest of  $D$ .
We will use the following test, where  $\Pi^? = \Pi_i^\prime$  is the null hypothesis and the alternative hypothesis is  $\Pi^? = \Pi_i$ . Let  $z = \mathrm{median}_{d\in D}(\Delta_{\mathcal{M}}(d,W_i))$ , estimated via a small number of samples from  $D$ . For Pick  $n_t$  datapoints from data sequence  $\Pi^7$ , and for each data point  $d\in \Pi^7$ , check if it's  $>z$ . Under the null hypothesis, the probability that each point passes this check is 0.5. Let the test statistic be  $t$ :
$$
t \left(\Pi^ {?} \right) = \sum_ {d _ {j} \sim \Pi^ {?} , j = 1, \dots , n _ {t}} \mathbb {I} \left(\Delta_ {\mathcal {M}} \left(d _ {j}, W _ {i}\right) > z\right) \tag {11}
$$
where  $\mathbb{I}$  is the indicator function. The value of  $t(\Pi_i')$ , the statistic under the null hypothesis, is distributed as a binomial with biased coin probability  $c = 1/2$  and  $n_t$  samples. However, we expect that  $t(\Pi_i)$  is a binomial with a larger  $c$ . To compute our confidence that  $W_i$  was trained using the data order  $\Pi_i$ , we can use a one-sided binomial hypothesis test, computing a  $p$ -value as  $1 - CDF_{binomial}(c = 1/2, n_t, k < t)$  where  $k$  is the value up to which to calculate the CDF. This statistic can be computed jointly across all checkpoints (requiring relatively few samples per checkpoint) to prove that the overall data ordering matches the one defined in Section 4.3.
Note that this test is similar to the "subtraction upper bound" heuristic from Section 4.2, with the key difference being that in this test we compare against the distribution of all training points  $D$  (since the counterfactual is a randomly selected subset of training data), whereas the subtraction test compares against points from the validation set  $D_v$  (since the counterfactual is that the points are never included in training). As an additional note, this same test can be generalized by replacing the median with a quantile, which may improve sample efficiency depending on the shape of the  $\Delta_{\mathcal{M}}$  distribution on  $\Pi_i$  vs.  $\Pi_i'$ .
D Verifier Objectives Table
Transcript Use-Case	Attacker Motivation	Definition of Defender Success
Check whether a model W was trained on data from a disallows distribution (e.g., relating to backdoors, cyberexploit generation, or enabling an undisclosed modality such as images).	A Prover wants to claim that W lacks a certain ability in order to avoid scrutiny, and does so by claiming W has only been trained on data from distribution D and not on distribution D’.	A test such that, given target weight checkpoint W, confirms that its training data did not include, in addition to a known number of data points n from a known distribution D, an additional kn training points from a different distribution D’.
Check whether a model W* was trained on greater than a certain number of data points, in case policy oversight targets the total training compute of a model (e.g. as part of compute usage reporting).	Underreport total training time to avoid triggering oversight.	A test such that, given a target weight checkpoint W* and claimed sequence of n data points II, detects whether the model was in fact trained on > kn data points, for some k > 1.
Check whether a model W was initialized without using weights obtained from previously-trained models.	A Prover might wish to start training using weights obtained from a previous training run, hiding the fact that more data or compute was used than reported, in order to avoid scrutiny, or to save compute by copying another’s work.	A test such that, given a desired initialization W0 (up to hidden unit permutations), makes it cryptographically hard to construct a transcript that results in W0 being an initialization compatible with the resulting transcript.
Check whether a model has a backdoor, i.e. an improbable input that yields a disallowed behavior.	An attacker might wish to hide capabilities, or give themselves unauthorized access to systems that will be gatekept by deployed versions of their models.	A test such that, given a transcript, allows reliable detection of backdoors through code or data audits.
Check whether a model was trained using at least a certain quantity of data, e.g., as part of a Proof-of-Learning meant to verify the original owner of a model, or to verify that certain safety-best-practice training was done.	A Prover may wish to save on compute costs by doing less training, or to prevent their model from being trained on required data.	A test such that, given a target weight checkpoint W* and a claimed sequence of n data points II, detects whether the model was in fact trained on < cn data points, for some c < 1.
Check whether a model was trained using a particular data-point.	A Prover may wish to train on copyrighted content, or uncurated datasets, or obfuscate which training data were used.	A test such that, given a transcript and a target datapoint x, detects whether the model was in fact trained on x.
E Hardness of Spoofing a Weight Initialization
To recap, by requiring that the Prover initialize a model's weights at a specific value in high-dimensional space  $W_0 \in \mathbb{R}^d$  drawn from a pseudorandom vector generator  $G_r$ , we seek to disallow a class of spoofing attacks based on the Prover hand-picking an initial weight vector  $\hat{W}_0$  that will
after training end up close to  $W_{f}$ , for example by picking an initialization that is already close to  $W_{f}$  (Attack 2 in [ZLD+22]).
The simplest setting in which defense is impossible, and the Prover can reliably find a random initialization that will converge to a given  $W_{f}$ , is in realizable linear models (models with only a single linear layer). Since their loss function is strongly convex, any initialization will converge to a neighborhood of the same final value  $W_{f}$ , making it straightforward to construct tweaked datasets with certified-random initializations that result in approximately the same final model. Another counterexample occurs when datasets have a single degenerate solution: it is possible to construct a 2-layer neural network with training data covering the input space and where all the labels are 0, such that the model always converges to a weight vector of all 0s, independent of initialization. We will focus our discussion on the usual case of multi-layer NNs with non-degenerate solutions, as described below.
Below, we will sketch an informal argument that for some radius  $r$ , for a fixed training data sequence  $\Pi$ , the probability that a training run initialized at a pseudorandomly-generated  $^{10}$  weight vector  $W_{0} = G_{r}(s)$  ends in a final weight vector  $W_{f}$  that is within distance  $r$  of a particular target vector  $A$ , is less than some small value  $\delta < \tilde{o}(1/poly(d))$ , where  $d$  is the dimension of the neural network. This means that a Prover would need to sample a super-polynomial (in  $d$ ) number of random seeds to find one that would, via training on  $\Pi$ , result in a fully-valid training transcript that ends close to the weight vector  $W_{f}$  from a previous training run with a different initialization, and therefore that it is exponentially hard to violate the "uniqueness" property from Section 3 if the Prover uses a certified random initialization.
To understand whether this is the case, we can examine the counterfactual claim: that independent of weight initialization, all NNs tend to converge to a small (polynomial) number of modes in weight space. This is indeed the case with linear regression: regardless of the initialization, given sufficient full-rank data all linear model training runs will converge to a neighborhood of the same loss minimum in weight-space. If this were also true for neural networks, then even a small number of randomly-sampled weight initializations would likely yield at least one weight initialization that, after training, converged to a mode close to the target  $A$  (assuming  $A$  is close to at least one mode, which is the case when  $A$  is the outcome of a previous training run  $W^{f}$ ). Yet, empirically, many works have found that large NNs converge to many different modes [AHS22, FDRC20].
The many modes of the NN loss landscape can be understood through permutation symmetries [AHS22]. Neural networks are equivariant ("equivariant" means that a function changes symmetrically under a group action) under specific permutations of their matrices' columns and rows. Nearly all neural networks have the following permutation symmetries: given a single hidden layer  $M_1\sigma(M_2(x))$  where  $M_1 \in \mathbb{R}^{a \times b}$ ,  $M_2 \in \mathbb{R}^{b \times c}$  and  $\sigma : \mathbb{R}^b \to \mathbb{R}^b$  is a nonlinearity, and given any permutation matrix  $F \in \mathbb{R}^{b \times b}$  (such that  $FZ$  permutes the rows of  $Z$ ), then by simple algebra  $M_1F^T\sigma(FM_2x) = M_1\sigma(M_2x)$  for all  $x$ . This means that for any set of successive NN matrices  $M_1, M_2$ , there are at least  $b!$  possible permutations with identical input output behavior. For a neural network  $W$  with  $k$  nonlinear layers and hidden dimension of each layer  $b$ , there could be  $k - 1$  different permutation matrices  $F_1, F_2, \ldots$ , and we denote to the operation of permuting the flattened weight vector  $W$  using a particular value of these  $F_s$  as  $P : \mathbb{R}^d \to \mathbb{R}^d$ . Each  $P$  is drawn from the overall set of valid permutations for a particular architecture  $P \in \mathbb{P}(M)$ , and we know that  $\| \mathbb{P}(M)\| = \Omega(2^{kb\log b})$ .
A second important property is that gradient descent is itself equivariant under the described permutations. Let  $R$  be the training operator, such that  $W_{f} = R(W_{0},\Pi)$  is the result of training initial weights  $W_{0}$  on a data sequence  $\Pi$ . Then it is true that  $\forall P \in \mathbb{P}(M)$ ,
$$
P (W _ {f}) = P \left(R (W _ {0}, \Pi)\right) = R \left(P (W _ {0}), \Pi\right) = W _ {f} ^ {p}
$$
where  $W_{f}^{p}$  is the result of training on the permuted initialization. This is simply a consequence of the fact that the gradient operator commutes with any constant matrix (including the permutation matrix), and that the training process  $R$  is comprised of repeated calls to the gradient operator, additions, and scalar multiplications (both of which also commute with the permutation matrix).<sup>12</sup>
<sup>10</sup>Assuming that  $s$  is chosen randomly, based on assumptions described in Section 4.3.
<sup>11</sup>We omit the inherent noise and hyperparameters inherent in  $R$  for brevity.
<sup>12</sup>It is in principle possible to construct optimizers for which this is not the case, but this should hold for all common gradient-based NN training optimizers.
Now, assume that the initialization function  $G_{r}$  is radially symmetric (as is the case with all common initialization schemes, e.g., those based on Gaussians), and therefore the probability that the initialization will start at  $W_{0}$  and  $P(W_{0})$  is the same for all  $P \in \mathbb{P}$ . Then the probability that the post-training final weights reach  $W_{f}$  or  $P(W_{f})$  is also the same. If we knew that  $\operatorname*{Pr}_{W_0 \sim G_r(s)}(\| W_f - P(W_f)\| > 2r) > 1 - \delta$  for some  $r$  and small  $\delta$ , then this derivation would tell us that there are many different weight-space modes into which training could converge, each of which is far apart from the others. (For convenience, let's refer to the number of such far-apart permuted modes as  $k$ .)
Again, our goal is to show that a random initialization is unlikely to converge after training to within a neighborhood around some vector  $A$ . Assume that  $B$  is one of these modes, and  $\| A - B \| < r$ . According to the assumption from the previous paragraph on the distance between post-training modes, for any second mode  $C$ , we know that  $\| C - B \| > 2r$  with high probability. By the triangle inequality, we know that:
$$
\begin{array}{l} \| A - C \| \geq \| C - B \| - \| A - B \| \\ > 2 r - r = r \\ \end{array}
$$
Therefore there is some minimum distance  $\| A - C \| > r$  between the target  $A$  and all other  $k$  disjoint modes (each associated with a permutation) of the post-training weight distribution. If the number of such far-apart permutations  $k$  is superpolynomial, then no polynomial number of weight initialization samples will result in a final model close to  $A$ .
However, this argument is predicated on a sometimes-invalid assumption: that there are superpolynomially-many permutations  $k = \omega(poly(d))$  of  $W_{f}$ , each at least a distance  $2r$  from each other. In the case of the counterexample from the beginning, where all initializations converge after training to the weight vector of all 0s, all such permutations are in fact equal, and therefore there is no such distance  $r$ . Instead, one may need to make an assumption about the non-degeneracy of the distribution of final weight vectors  $W_{f}$ , such that permutations of these weight vectors are far apart from each other. We leave analysis of which assumptions fulfill this property as future work. Note that for any specific training transcript which includes a specific  $W_{f}$ , the distribution of distances of permutations of  $W_{f}$  can be estimated empirically by manually permuting  $W_{f}$ 's matrices.
For the GPT-2 Experiments we use a cosine learning rate schedule that decays by a factor of 10x by the end of training, with a linear warmup of 2000 steps to a peak learning rate of 0.0006. For the Pythia evaluation experiments, we choose checkpoints from 3 contiguous blocks out of 144 checkpoints: early (first 19 checkpoints), mid (checkpoints at step 62000 to 80000), and late (last 19 checkpoints).
G More memorization plots
In the following subsections, we plot memorization  $\mathcal{M}$ , fraction of points with  $\Delta_{\mathcal{M}}$  above the median, and fraction of points with  $\Delta_{\mathcal{M}}$  below the 10th percentile. For GPT-2, we use  $100\%$  of the data to generate Figures 1, 2, and 3, while for Pythia, we use  $10\%$  of the data to generate Figure 2. In this section, we show results for smaller sampling rates to highlight that with  $1\%$ , or sometimes even  $0.1\%$  of the original data, we can still observe the memorization effect.
From Pythia 70M results (Subsections G.1, G.2, and G.3) we can see that as training progresses, the memorization effect becomes less pronounced, such that with a smaller data sampling rate, less of the diagonal get highlighted (Figures 11, 18, 25), and the histograms are closely overlapping (Figure 32) for the last 18 checkpoints. At the same time, we observe that as the model size increases the memorization effect becomes clearer, even with  $0.1\%$  data sampling rate. In fact, for the 1B-parameter Pythia model, the memorization effect is still clear for the last few checkpoints (Figures 14, 21, 28, and 35) unlike the 70M-parameter case.
<sup>13</sup>If this is untrue for all modes  $B$ , then by definition there is no initialization that leads close to  $A$ , which satisfies our original objective of bounding the probability of the final weights converging to a neighborhood of  $A$ .
G.1 Memorization
Figure 8: Memorization plots for GPT-2 with different sampling rates.
GPT-2 120M
Figure 9: Memorization plots for the first 18 checkpoints of Pythia (70M) with different sampling rates.
Pythia 70M Early
Figure 10: Memorization plots for checkpoints near the middle of Pythia (70M) with different sampling rates.
Pythia 70M Mid