SlowGuess/NeurIPS_2023_OCR huggingface dataset:
in this experiment. At depths less than 15,  $C_T$  is a positive value, indicating the training samples are no closer to the generated samples than they are to ones from the test set (are not memorized), while at depths greater than 15,  $C_T$  becomes increasingly negative, indicating a stronger detection of memorization. AuthPct starts from close to  $100\%$ , meaning all samples are authentic, and decreases to  $20\%$  at a depth of 41, indicating that generated samples are closer to training samples than expected. Since a depth of zero means no conditioning on individual training images (except for the training of the original model), the result confirms the effectiveness of the proposed method for this particular point. As the depth increases, AuthPct correctly decreases the percentage of authentic samples in a rate that is almost equal to the decrease in  $C_T$ . FLS-POG has the correct trend as it increases as samples become more memorized. However, the rate of the increase is minimal, does not reflect the level of memorization seen at a depth of 41 (as seen in the images training samples which are almost exactly reproduced at this depth), and is not comparable to the change in the other two metrics. From this controlled experiment we conclude that, in the absence of mode collapse/mode shrinkage when training samples are uniformly memorized,  $C_T$  and AuthPct can flag memorization in practice while FLS-POG does not flag memorization strongly enough.
D.6.2 Memorization metric quality
With a better understanding of AuthPct, FLS-POG, and the  $C_T$  score, we calculate them on the sets of CIFAR10 generated images and compare to the measured memorization ratios determined through the analysis above. Results for the DINOv2 representation space were shown in Section 5.2, while here we show the scores for every encoder in Figure 24. We also include our modification to the  $C_T$  score which swaps the roles of the training and test samples (details and motivation for the modified  $C_T$  score follow in Appendix D.6.3). We remind the reader that higher AuthPct,  $C_T$  score, and modified  $C_T$  score are meant to indicate less memorization, whereas lower FLS-POG is meant to indicate less memorization.
Here we detail the characteristics of each encoder. For Inception, we see that a higher memorization ratio seems to correctly correlate with lower AuthPct and  $C_T$  score. However there is no discernible pattern for FLS-POG and  $C_T$  modified. Meanwhile, the SwAV encoder seems to provide the correct directionality for all memorization metrics. CLIP produces no discernible trend for AuthPct and FLS-POG, a slightly accurate trend for  $C_T$ , and a strongly accurate trend for modified  $C_T$ . MAE is similar to CLIP except with the roles of FLS-POG and  $C_T$  perhaps flipped. data2vec is somewhat similar to SwAV with the correct directionality – particularly for  $C_T$  score – but with a slightly weaker trend for the other metrics. Lastly, DINOv2 shows a correct correlation for the modified  $C_T$  score, but no strong trends for the others. While the performance of each encoder shows a large variation, it is important to note that we illustrate clear issues with the memorization metrics in the next section (D.6.3). We find that modified  $C_T$  score performs the best in this section, with  $C_T$  score next, then AuthPct, then FLS-POG last. This agrees with our previous experiment that shows FLS-POG is the weakest detector of memorization.
D.6.3 Low-dimensional experiments for memorization metrics
Table 12: Memorization metrics on synthetic datasets
Metric	True Distribution	Shrinkage	Memorized	Underfit 1	Underfit 2	Underfit 3
AuthPct	41.20	20.00	0.00	46.60	67.20	77.30
CT Score	-0.23	-16.14	-25.26	7.16	15.22	18.42
CT Mod.	0.05	-0.86	-16.71	-0.62	0.23	1.37
FLS-POG	60.10	60.00	89.70	59.00	56.30	55.20
While the previous subsections have confirmed that  $C_T$  and AuthPct can flag memorization in ideal scenarios (Section D.6.1) and seemingly show appropriate trends in some representation spaces (Section D.6.2), it is not clear that they are explicitly flagging memorization over other forms of overfitting. Curiously, the metrics computed against the test set resemble those against the training set, which the models cannot have memorized outside of data-leakages between the construction of the train and test sets. While we note that such leakages do occur for CIFAR10 [6], in the following toy scenario we demonstrate the inability of the metrics to separate types of overfitting from memorization.
Figure 24: Memorization ratio vs. metric value for each metric out of AuthPct, FLS-POG,  $C_T$ , and the modified  $C_T$ , evaluated on the models from Figure 19. The same metric values computed against the test set are shown with lower opacity.
To understand the impact of various modelling phenomena on memorization metrics we design a simple 2-dimensional experiment. As a ground truth distribution, we take a uniform mixture of 5 Gaussian components. Each component's mean is sampled from a unit diagonal Gaussian distribution, while each component's covariance matrix is diagonal with diagonal elements sampled from a uniform distribution over the interval [0.01, 0.09]. From this distribution we sample 1000 training and 1000 test points.
Figure 25: Toy datasets generated for the purpose of investigating memorization metrics.
We then generate 1000 points from various "models" from which to measure memorization:
- A perfect model with zero memorization, represented by independent samples from the same ground truth distribution.
- A model experiencing total mode shrinkage: it uniformly samples a mean from each Gaussian in the mixture.
- A model that has perfectly memorized the training data: it amounts to randomly sampling from the training data with replacement.
- Three "underfit" models, which in fact have learned the true distribution with upscaled standard deviation for each component of the Gaussian mixture:  $1.5\sigma$ ,  $3\sigma$ , and  $4.5\sigma$ .
The training, test, and generated distributions are shown in Figure 25. We evaluate metrics on these datasets and report the values in Table 12.
FLS-POG seemingly performs correctly for detecting memorization in these regimes - with the memorized model receiving the highest score of  $80\%$  and the other models receiving a score hovering around  $60\%$ . However, a value of  $60\%$  on both the true and mode shrinkage examples indicates that it poorly flags mode shrinkage, one of its claimed usages [53], and the relative change of  $5\%$  from underfit to the true distributions is on the order of the change seen in the VDVAE controlled experiment, indicating that such relative changes can have multiple meanings. AuthPct indicates memorization  $(< 50\%)$  for the mode shrinkage model, the perfect model, and the slightly underfit model. From this, we gather that AuthPct incorrectly flags well-trained models that produce samples on the true manifold.
We also find that the  $C_T$  score performs incorrectly here: the score of -16.14 it assigns to the mode shrinkage model – well below 0 – has incorrectly flagged memorization. We can see why by looking closer at the metric: the  $C_T$  score compares how much more often (i) the distance from a generated data point to a training data point is lower than (ii) the distance from a test data point to a training data point, and reports a highly negative score if (i) is often lower than (ii). In this case, we would expect our generated, mode-shrunken samples to be much closer to the training data than the test data is to the training data, which agrees with what we observe. The issue here boils down to a lack of symmetry between test and training set: the test set is only used for one of the comparisons, whereas the training set is used in both distance checks. The generated dataset is also only used once. We address this asymmetry by proposing a modification to the  $C_T$  score – referred to as the modified  $C_T$  score – wherein we essentially swap the roles of the training and the generated data within the base  $C_T$  score. The modified  $C_T$  score thus compares how much more often (i) the distance from a generated data point to a training data point is lower than (ii) the distance from a generated data
Figure 26: ImageNet samples for generative models (rows) sorted by error rate from our human subject experiments. Classes from left to right: great white shark, English Springer, comic book, and volcano.
point to a test data point, reporting a highly negative score if  $(i)$  is often lower than  $(ii)$ . To fully summarize and perhaps visualize our change, consider the  $C_T$  score as a function from training data  $T_1$ , generated data  $G$ , and test data  $T_2$  to the real numbers, represented as  $C_T(T_1, G, T_2)$ . Then, the modified  $C_T$  score - call it  $C_{T,M}$  - can be represented as  $C_{T,M}(T_1, G, T_2) \equiv C_T(G, T_1, T_2)$ . We see that our change has produced the desired behaviour, as now the mode shrinkage dataset is not detected as memorized by  $C_{T,M}$ .
From this discussion we gather that the  $C_T$  score incorrectly flags mode shrinkage, and AuthPct incorrectly flags well-trained models. Recall that none of the models except for the memorized one are actually derived from the test set. These experiments explain why AuthPct and  $C_T$  seem to score models in a way that is almost invariant to the training set in Figure 7.
D.7 Model samples
Figures 26, 27, 29, and 28 show image samples from the 41 generative models across the four datasets.
Figure 27: CIFAR10 samples for generative models (rows) sorted by error rate from our human subject experiments. Non class-conditional models show a random selection of generated samples while class-conditional models show classes airplane, automobile, bird, and dog.
Figure 28: LSUN Bedroom samples for generative models (rows) sorted by error rate from our human subject experiments.
Figure 29: FFHQ samples for generative models (rows) sorted by error rate from our human subject experiments.
E  $\mathbf{FD}_{\mathrm{DINOv2}}$  scores leaderboard
Table 13 shows our considered metrics across models and datasets.
F Author contributions
Table 14 lists individual contributions of the authors (in alphabetical order).
Table 14: Author contributions
AC	BR	ET	GL	GS	JC	RH	VV	YS	ZL
Conceived of ideas for project initially	X	X	X	X
Conceived human experiments	XX	X	X	XX
Created, performed, & analyzed human exp.	X	XX
Trialed human experiments	X	X	X	X	X	X	X	X	X	X
Generated CIFAR10 datasets	X	X	X	X
Generated ImageNet datasets	X	X	X	X	X
Generated FFHQ datasets	X	X	X	X
Generated LSUN-Bedroom datasets	XX	X
Implemented encoders	X	X	X	X	X
Grad-CAM encoder understanding	X	X	XX	X
Designed metrics	X	X	X	X	X
Implemented metrics	X	X	X	X	X	XX
Memorization design	X	X	X	X	X	X
Memorization implementation	X	X	X	X
Diversity	X	X	X
Other experiments	X	X
Structured Codebase	XX	X	X	X	XX
Prepared figures	X	X	X	X	X	X	X
Analyzed data	X	X	X	X	X	X	X	X
Wrote paper	X	X	X	XX	XX	XX	X	X	X	X
Managed the project	X	XX