SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In addition, getting the likelihood assigned by each Gaussian to each point requires computing the distance between each pair  $(\mathbf{x}_i^{\mathrm{gen}},\mathbf{x}_i^{\mathrm{train}})$  which is  $O(n^{2})$  and time-consuming. Also, as the distances and dimensions are large (at least for our high-dimensional experiments), the exponentiation and summations were often numerically unstable. To address this, we:
- Compute the  $O(n^{2})$  distance matrix once and store it so as not to recompute it for each step of the optimization procedure.
- Optimize the log variances instead of the variances themselves
- Convert  $\sigma_{j}^{-d}$  to  $\exp(-d\log(\sigma_{j}))$  to be able to take advantage of a numerically stable logsumexp.
From plotting the loss, the variances almost always converged in short order (usually less than 10 steps). While we have no guarantee that these were global minima, when there were exact copies or close to exact copies, 1 would recover very low log variances, as shown in 1.
E.1 Transforms
To assess both quality and overfitting, we use standard torchvision [maintainers and contributors, 2016] image transforms described below:
- Posterize: Posterize the image to 5 bits (we found 5 was the lowest we could go while the difference still being mostly imperceptible).
- Resize: Resize the image to the size accepted by InceptionV3 using bicubic interpolation which has been shown to be problematic [Parmar et al., 2022].
- Color Distort: Apply the color jitter transform described in [maintainers and contributors, 2016] with default parameters.
- Elastic transform: Apply the elastic transform described in [maintainers and contributors, 2016] with default parameters.
- Blur: Gaussian blur with a (5, 5) kernel. For the light blur, we use a  $\sigma$  of 0.5. For the heavy blur we use  $\sigma = 1.4$ . For the heatmap, the blur intensity corresponds to the value of  $\sigma$
- Center crop x: Center crop the image to  $(x, x)$  and fill the rest with black.
- JPG x: Convert image to a JPG of quality x.
E.2 Generated samples
For all experiments, we have updated our paper to use the samples provided by Stein et al. [2023] who have generously open-sourced an excellent set of samples from a wide variety of SOTA models on various datasets.
We now prove our Proposition 1 from the main paper.
Proof. We begin by showing that  $\hat{\sigma}_l^2 = O(\delta_l^2)$  for  $\delta_l$  small enough.
We define the variance vector  $\hat{\sigma}$  to be the maximizer of the following equation
$$
\hat {\sigma} ^ {2} \in \arg \max  _ {\sigma^ {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \log \left(\frac {1}{m} \sum_ {j = 1} ^ {m} \frac {1}{(\sqrt {2 \pi} \sigma_ {j}) ^ {d}} \exp \left(\frac {- \| \varphi \left(\mathbf {x} _ {j} ^ {\text {g e n}}\right) - \varphi \left(\mathbf {x} _ {i} ^ {\text {t r a i n}}\right) \| ^ {2}}{2 \sigma_ {j} ^ {2}}\right)\right). \tag {7}
$$
Without loss of generality, we consider the following shifted and rescaled objective function with the same argmax as Eq. 7
$$
f (\sigma) := \sum_ {i = 1} ^ {n} \log \left(\frac {1}{m} \sum_ {j = 1} ^ {m} \frac {1}{\sigma_ {j} ^ {d}} \exp \left(\frac {- D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right)\right). \tag {8}
$$
Then, for any  $i = 1,\ldots ,n$  , we have
$$
\max  _ {j} \sigma_ {j} ^ {- d} \exp \left(\frac {- D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right) \leq \sum_ {j = 1} ^ {m} \sigma_ {j} ^ {- d} \exp \left(\frac {- D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right) \leq m \max  _ {j} \sigma_ {j} ^ {- d} \exp \left(\frac {- D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right) \tag {9}
$$
and using the fact that log is an increasing function, we have that for any  $\sigma$
$$
g (\sigma) := \sum_ {i = 1} ^ {n} \max  _ {j} \left(- d \log \sigma_ {j} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right) \leq f (\sigma) \leq g (\sigma) + n \log (m) \tag {10}
$$
Now, let  $\sigma^{*} \in \arg \max g(\sigma)$  and take  $l \in \{1, \dots, m\}$ . We now show that there exists a  $\bar{\delta}$  such that for  $\delta_{l} < \bar{\delta}$ ,  $\sigma_{l}^{*} = \frac{\delta_{l}}{\sqrt{d}}$  and appears exactly once in the sum  $g(\sigma^{*})$ .
Suppose  $\sigma_{l}^{*}$  appears in  $k$  terms (WLOG we can assume they are the first  $k$  terms). We can then break down  $g(\sigma^{*})$  as follows,
$$
g _ {k} \left(\sigma^ {*}\right) = \sum_ {i = 1} ^ {k} \left(- d \log \sigma_ {l} ^ {*} - \frac {D _ {i l} ^ {2}}{2 \sigma_ {l} ^ {* 2}}\right) + \sum_ {i = k + 1} ^ {n} \max  _ {j \neq l} \left(- d \log \sigma_ {j} ^ {*} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {* 2}}\right) \tag {11}
$$
$$
C := \sup  _ {\sigma} \sum_ {i = k + 1} ^ {n} \max  _ {j \neq l} \left| - d \log \sigma_ {j} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}} \right| <   + \infty . \tag {12}
$$
We then examine the following three cases:
-  $k = 0$ : The sum in  $g(\sigma^{*})$  only consists of  $j \neq l$ .
$$
g _ {0} \left(\sigma^ {*}\right) = \sum_ {i = 1} ^ {n} \max  _ {j \neq l} \left(- d \log \sigma_ {j} ^ {*} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {* 2}}\right) \tag {13}
$$
which is bounded by Eq. 12.
-  $k = 1$ : Eq. 11 is maximized with respect to  $\sigma_{l}^{*}$  for  $\sigma_{l}^{*} = \frac{D_{1l}}{\sqrt{d}}$ . Plugging in to Eq. 11
$$
g _ {1} \left(\sigma^ {*}\right) = - d \log \left(D _ {1 l}\right) - \frac {d}{2} (\log (d) + 1) + \sum_ {i = 2} ^ {n} \max  _ {j} \left(- d \log \sigma_ {j} ^ {*} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {* 2}}\right) \tag {14}
$$
Specifically, if the only term  $\sigma_{l}$  appears in is the one with  $D_{1l} = \delta_l$
$$
\hat {g} _ {1} \left(\sigma^ {*}\right) = - d \log \left(\delta_ {l}\right) - \frac {d}{2} (\log (d) + 1) + \sum_ {i = 2} ^ {n} \max  _ {j} \left(- d \log \sigma_ {j} ^ {*} - \frac {D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {* 2}}\right) \tag {15}
$$
where the term  $-d\log (\delta_l)$  and thus  $\hat{g}_{k = 1}(\sigma^{*})$  can be made arbitrarily large as  $\delta_l\to 0$ . In contrast, if  $\sigma_{l}^{*}$  appears for  $D_{1l} > \delta_{l}$  we get a bounded  $g_{1}$  (independently of  $\delta_l$ ).
-  $k > 1$ : Eq. 11 is maximized with respect to  $\sigma_{l}^{*}$  for  $\sigma_{l}^{*} = \sqrt{\frac{\sum_{i=1}^{k} D_{il}^{2}}{kd}}$ . Plugging in to Eq. 11
$$
\begin{array}{l} g _ {k} \left(\sigma^ {*}\right) = \sum_ {i = 1} ^ {k} \left(- d \log \left(\sqrt {\frac {\sum_ {i = 1} ^ {k} D _ {i l} ^ {2}}{k d}}\right) - \frac {D _ {i l} ^ {2}}{2 \frac {\sum_ {i = 1} ^ {k} D _ {i l} ^ {2}}{k d}}\right) (16) \\ = - k d \log \left(\sqrt {\frac {\sum_ {i = 1} ^ {k} D _ {i l} ^ {2}}{k d}}\right) - \frac {k d}{2} (17) \\ \leq - k d \log \left(\sqrt {\frac {(k - 1)}{k d}} \tilde {\delta} _ {l}\right) - \frac {k d}{2} (18) \\ \end{array}
$$
and is thus bounded independently of  $\delta_l$ .
Thus, we conclude that, for  $\delta_l$  small enough,  $\sigma_l^* = \frac{\delta_l}{\sqrt{d}}$  since  $\hat{g}_1(\sigma^*) > \{g_0(\sigma^*), g_1(\sigma^*), g_k(\sigma^*)\}$ .
Now, consider  $\tilde{\sigma}$  such that  $\tilde{\sigma}_l = \sigma_l^* /\sqrt{1 - c}$  and  $\tilde{\sigma}_j = \sigma_j^*$  otherwise. We now look at  $g(\tilde{\sigma})$ . From the above
$$
g (\tilde {\sigma}) = g \left(\sigma^ {*}\right) + \frac {d}{2} (\log (1 - c) + c) \tag {19}
$$
as long as  $\tilde{\sigma}_l$  is found in a single term of the summation in  $g(\tilde{\sigma})$  (i.e.  $k = 1$ ). If not, with the same argument as above,  $g(\tilde{\sigma})$  is bounded above by a constant  $U$  independent of  $\delta_l$  and  $c$ . Thus  $\exists 0 < \tilde{\delta} < \bar{\delta}$  such that  $\forall \delta_l < \tilde{\delta}$ , we have  $U + n\log m \leq g(\sigma^*)$ . Thus, in both cases  $\forall \delta_l < \tilde{\delta}$  and for all  $c$  such that  $\frac{d}{2} (\log (1 - c) + c) < -n\log m$ , we have
$$
\begin{array}{l} f (\tilde {\sigma}) \leq g (\tilde {\sigma}) + n \log m (20) \\ \leq \max  \left[ U, g \left(\sigma^ {*}\right) + \frac {d}{2} (\log (1 - c) + c) \right] + n \log m (21) \\ <   \max  [ U, g (\sigma^ {*}) - n \log m ] + n \log m (22) \\ \leq g \left(\sigma^ {*}\right) (23) \\ \leq f (\hat {\sigma}). (24) \\ \end{array}
$$
Since  $\frac{d}{2} (\log (1 - c) + c)$  is decreasing in  $c$  (for  $c > 0$ ) and  $\tilde{\sigma}_l$  is increasing in  $c$ , we must have  $\hat{\sigma} < \tilde{\sigma}_j = \frac{\delta_j}{d\sqrt{1 - c}}$  where  $1 - c = \exp \left(-\frac{2n + 1}{d}\log (m)\right)$ . Thus,  $\hat{\sigma} = O(\delta_j)$  for the case  $\delta_l < \tilde{\delta}$ .
For the case  $\delta_l \geq \bar{\delta}$ , we first show that  $\hat{\sigma}_l$  is bounded ( $\hat{\sigma}_l \leq M$ ). Denote the individual likelihood terms of  $f(\sigma)$  as follows
$$
L _ {i j} \left(\sigma_ {j}\right) := \frac {1}{\sigma_ {j} ^ {d}} \exp \left(\frac {- D _ {i j} ^ {2}}{2 \sigma_ {j} ^ {2}}\right). \tag {25}
$$
Then  $\forall i$ , we have that
$$
L _ {i l} \left(\sigma_ {l}\right) \leq \frac {1}{\sigma_ {l} ^ {d}} \underset {\sigma_ {l} \rightarrow \infty} {\rightarrow} 0 \tag {26}
$$
Thus, since  $f(\sigma)$  is increasing in the  $L_{ij}(\sigma)$ , it suffices to show there is  $\sigma_l \leq M$  such that all  $L_{il}(\sigma_l)$  are positive. Taking  $\sigma_l = 1$ , we have that  $\forall i: L_{il}(1) \geq e^{-\bar{D}^2 / 2} > 0$ . Thus we can take  $M = e^{\frac{D^2}{2d}}$ . Hence, for the case  $\delta_l \geq \tilde{\delta}$
$$
\begin{array}{l} \hat {\sigma} _ {l} \leq \frac {\delta_ {l}}{\bar {\delta}} \hat {\sigma} _ {l} \\ \leq \frac {M}{\bar {\delta}} \delta_ {l}. \\ \end{array}
$$
Finally  $\hat{\sigma}_l$  is  $O(\delta_l)$  by taking  $C = \max \left\{\frac{1}{d\sqrt{1 - c}},\frac{M}{\bar{\delta}}\right\}$ .
G Feature spaces
Feature spaces play a very important role in our metric and similar metrics (FID, Precision/Recall, etc.). While those produced by InceptionV3 have proved useful, there is work illustrating some potential issues, motivating our switch to DINOv2. As such, while the resulting feature psace has proved adequate for our experiments, it is likely that FLD could provide an even better assessment of generated samples with better feature embeddings. Additionally, while this work has solely focused on the evaluation of generative models in the image domain, there is nothing about FLD indicating it could not be applied to other domains (using appropriate feature embeddings) and we leave this as a fruitful future area of research.
H Broader Impact
Generative modeling has the potential to create copies of people's identities or artistic styles without their consent, which can have negative impacts on individuals and communities. While evaluation metrics can ensure the generalizability of generative models, they should not be employed to justify copying an artist's style without their permission. The concept of copyrighting an artistic style is intricate, and while some may argue its challenges and ambiguities, it is vital to acknowledge the impact on the artist whose style is being reproduced. Respecting the rights of individuals and communities whose data is used in creating generative models is crucial to avoid harm or exploitation. As the field of generative modeling progresses, it is imperative to engage in ongoing discussions regarding the ethical implications of these technologies and establish guidelines for their responsible application. Additionally, if evaluation metrics fail to consider the generation of harmful or unethical content, negative societal consequences can emerge. Therefore, it is crucial to consider a diverse range of evaluation metrics that encompass various aspects of model performance while aligning with ethical and societal considerations.