SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 11: Memorization plots for the last 18 checkpoints of Pythia (70M) with different sampling rates.
Pythia 70M Late
Figure 12: Memorization plots for checkpoints near the middle of Pythia (410M) with different sampling rates.
Figure 13: Memorization plots for checkpoints near the middle of Pythia (1B) with different sampling rates.
Figure 14: Memorization plots for checkpoints near the end of Pythia (1B) training with different sampling rates.
G.2 Fraction of Samples Above 50th Percentile
Figure 15: Fraction of samples above the 50th percentile for GPT-2 with different sampling rates.
Figure 16: Fraction of samples above the 50th percentile for the first 18 checkpoints of Pythia (70M) with different sampling rates.
Figure 17: Fraction of samples above the 50th percentile for Pythia (70M) with different sampling rates.
Figure 18: Fraction of samples above the 50th percentile for the last 18 checkpoints of Pythia (70M) with different sampling rates.
Figure 19: Fraction of samples above the 50th percentile for Pythia (410M) with different sampling rates.
Figure 20: Fraction of samples above the 50th percentile for Pythia (1B) with different sampling rates.
Figure 21: Fraction of samples above the 50th percentile for checkpoints near the end of Pythia (1B) training with different sampling rates.
G.3 Fraction of Samples Below 10th Percentile
Figure 22: Fraction of samples below 10th percentile for GPT-2 with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
Figure 23: Fraction of samples below 10th percentile for the first 18 checkpoints of Pythia (70M) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
Figure 24: Fraction of samples below 10th percentile for Pythia (70M) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
Figure 25: Fraction of samples below 10th percentile for the last 18 checkpoints of Pythia (70M) with different sampling rates.
Figure 26: Fraction of samples below 10th percentile for Pythia (410M) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
Figure 27: Fraction of samples below 10th percentile for Pythia (1B) with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
Figure 28: Fraction of samples below 10th percentile for checkpoints near the end of Pythia (1B) training with different sampling rates. White boxes occur whenever the number of samples falling below the 10th percentile is 0.
G.4 Memorization Delta Histograms
Figure 29: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from GPT-2 training.
Figure 30: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are the first 18 from Pythia (70M) training.
Figure 31: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (70M) training.
Figure 32: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are the last 18 from Pythia (70M) training.
Pythia 410M Mid
Figure 33: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (410M) training.
Pythia 1B Mid
Figure 34: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the middle of Pythia (1B) training.
Figure 35: Each subplot compares two histograms of  $\Delta_{\mathcal{M}}$ , one for  $\Delta_{\mathcal{M}}$  resulting from the checkpoint evaluated on its most recent data segment (diagonal), and one for the checkpoint evaluated on the validation set. All checkpoints are from near the end of Pythia (1B) training.
H More attack plots
H.1 Interpolation Attack
We repeat the interpolation attack experiment in the main body of the paper, with the 1B-parameter Pythia model and observe from Figure 36 that indeed the interpolated checkpoints fail our memorization tests.
Figure 36: Simulating an interpolation attack by training a Pythia (1B) model until the 67th checkpoint, and then linearly-interpolating to the 80th checkpoint. On the left, we show that an attacker can carefully choose interpolation points to mask any irregularities in validation loss. (The green line perfectly overlaps with the blue line.) Nonetheless, on the right, we see a clear signature in the memorization plot, computed using only  $1\%$  of data: the typical memorization pattern along the diagonal does not exist for the interpolated checkpoints. For each row corresponding to a data segment  $\Pi_{i}$ , a box marks the maximal- $\mathcal{M}$  checkpoint. The box is red if the checkpoint is a match  $W_{i}$ , and magenta if there is no match and the test fails  $W_{j \neq i}$ .
H.2 Data Subtraction Attack Tests
In the following subsections, we plot the subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  with varying values of  $p$ , for different subtraction rates. We observe that for big enough models  $\lambda$  is a tight upper-bound when no subtraction has happened. For Pythia with 70M parameters, our smallest model,  $\lambda$  does not provide a tight upper-bound. However, for GPT-2 with 124M parameters, Pythia with 410M parameters, and Pythia with 1B parameters,  $\lambda$  provides a tight upper-bound.
For the 1B-parameter Pythia model, we further plot the upper-bound heuristic for varying values of the checkpoint interval (number of training steps between each checkpoint). From Figure 41, we observe that even though  $\lambda$  increases as the interval increases, it is still a good upper-bound ( $\sim 0.05$  for a checkpoint interval of 5000 steps) for  $p = 0.1$  and  $p = 0.2$ . This means that we can save checkpoints less frequently and still use the heuristic to detect data subtraction.
H.2.1 GPT-2
Figure 37: The subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  is robust to different values of  $p$  especially for the full data case. The heuristic was computed using just  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
H.2.2 Pythia (70M)
Figure 38: The subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  does not provide a good upper bound for a smaller model (Pythia with 70M parameters). The heuristic was computed using  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
H.2.3 Pythia (410M)
Figure 39: The subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  is a good upper bound for a big enough model (Pythia with 410M parameters). The heuristic was computed using  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
H.2.4 Pythia (1B)
Figure 40: The subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  is a good upper bound for a bigger model (Pythia with 1B parameters). The heuristic was computed using  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
Figure 41: Even though the subtraction-upper-bound heuristic  $\lambda (\Pi_i,p,W_i)$  increases as number of steps between checkpoints increases, it still provides a good upper bound for  $p = 0.1$  and  $p = 0.2$ . The heuristic was computed using  $1\%$  of training data, across 20 random seeds, with dashed lines showing the true subtraction rate.
We intend this work to be a step towards meaningful and transparent public oversight of large AI systems, especially those with capabilities whose irresponsible use could significantly harm the public. Our protocol is a sketch of a technical framework for a system by which AI developers can prove properties of their training data, and may thereby enable the effective enforcement of a broader set of policies than those solely relying on querying models "black-box". While enabling many possible positive rules, this could also be misused by coercive states to detect and enforce harmful restrictions on beneficial AI development. However, in most cases, such authoritarian states would already have a means for policing domestic AI developers' behavior, and verification tools demanding so much cooperation from the Prover are unlikely to meaningfully increase existing surveillance powers. Another issue is that requirements for complying with monitoring and enforcement tend to favor large companies, for whom the cost of compliance can more easily be amortized. This motivates efforts to keep verification schemes simple, flexible and cheap.
We hope that this protocol can also be useful for verifying agreements between untrusting countries. The protocol itself does not provide a means for identifying that an AI model was developed in the first place unless it is disclosed. In this sense, it more closely parallels a process for an AI-developing country to allow its counterpart to retroactively inspecting a developed system (paralleling the New START treaties' inspections of nuclear launchers), rather than to proactively detect when a new system is developed (paralleling the IAEA's monitoring of the process of uranium enrichment).
Because our protocol supports multiple independent auditors reviewing the same transcripts, we hope that these tools will support the development of trust between competing companies and countries. Ultimately we hope such protocols will support the development of a larger governance ecosystem representing many parties.