Linear combinations of latents in generative models: subspaces and beyond
Authors: Erik Bodin
Alexandru I. Stere
Dragos D Margineantu
Carl Henrik Ek
Henry Moss
Venue: ICLR 2025 Poster (2025)
Keywords: generative models
diffusion models
latent space interpolation
latent subspaces
latent representations
flow matching
vae
Primary area: generative models
TL;DR: We show how latent subspaces can be defined in e.g. diffusion models, yielding expressive low-dimensional representations.

Abstract: Sampling from generative models has become a crucial tool for applications like data synthesis and augmentation. Diffusion, Flow Matching and Continuous Normalising Flows have shown effectiveness across various modalities, and rely on latent variables for generation. For experimental design or creative applications that require more control over the generation process, it has become common to manipulate the latent variable directly. However, existing approaches for performing such manipulations (e.g. interpolation or forming low-dimensional representations) only work well in special cases or are network or data-modality specific. 
We propose Latent Optimal Linear combinations (LOL) as a general-purpose method to form linear combinations of latent variables that adhere to the assumptions of the generative model. As LOL is easy to implement and naturally addresses the broader task of forming any linear combinations, e.g. the construction of subspaces of the latent space, LOL dramatically simplifies the creation of expressive low-dimensional representations of high-dimensional objects.

Published as a conference paper at ICLR 2025
LINEAR COMBINATIONS OF LATENTS IN GENERATIVE MODELS: SUBSPACES AND BEYOND
Erik Bodin!
Alexandru Stere*
Dragos D. Margineantu
Carl Henrik Ek
Henry Moss
University of Cambridge
21 Lancaster University
Karolinska Institutet
Boeenng Commercial Airplanes
Boeing Al
ABSTRACT
Sampling from generative models has become crucial tool for applications like data synthesis and augmentation. Diffusion, Flow Matching and Continuous Nor- malising Flows have shown effectiveness across various modalities, and rely on latent variables for generation. For experimental design or creative applications that require more control over the generation process, it has become common manip- ulate the latent variable directly. However, existing approaches for performing such manipulations (e.g. interpolation forming low-dimensional representations) only work well in special cases or are network or data-modality specific. We propose Latent Optimal Linear combinations (LOL) linear combinations of latent variables that adhere to the assumptions of the gener- ative model. As LOL is easy to implement and naturally addresses the broader task of forming any linear combinations, e.g. the construction of subspaces of the latent space, LOL dramatically simplifies the creation of expressive low-dimensional representations of high-dimensional objects.
purppose method form
INTRODUCTION
Generative models are cornersoone of machine learning, with diverse applications including image synthesis, data augmentation, and creative content generation. Diffusson models (Ho et al. 2020 Song et al. 2020a by have emerged as a particularly effective approach to generative modeling for various modalities, such as for images (Ho et al. 2020), audio (Kong et al. (020),, video (Ho et al.) 2022), and 3D models (Luo & Hu 2021). J. A yet more recent approach to generative modelling is Flow Matching (Lipman et 2022). built upon Continuous Normalising Flows (Chen et al. 2018 that generalises diffusion to allow for different probability paths between data and latent distribution,
e.g. defined through optimal transport (Gulrajani al. 2017
Villani et 2009).
As well as generation, these models allow inversion where, by running the generative procedure in the opposite direction, data objects can be transformed deterministically into a corresponding realisation in the latent space. Such invertible connections between latent and data space provide a convenient mechanism for controlling generated objects by manipulating their latent vectors. The most common manipulation is to attempt semantically meaningful interpolation of two generated objects (Song et al. 2020a b) Luo Hu 202T) by interpolating their corresponding latent vectors. However, the optimal choice of interpolant remains open question, with simple approaches like linear interpolation leading to intermediates for which the model fails to generate plausible objects.
White 2016) argues that poor quality generation under linear interpolation is due to mismatch between the norms of the intermediate vectors and those of the Gaussian vectors that the model has been trained expect. Indeed, it is well-known that the squared norm of D-dimensional unit Gaussian follows the chi-squared distribution. Consequently, likely samples are concentrated a tight annulus around a radius vD - a set which is not closed under linear interpolation. Because of this common to rely instead spherical interpolation (Shoemake 1985) (SLERP) that maintains similar

---

Published as a conference paper at ICLR 2025
norms the endpoints. Motivated by poor performance of interpolation between the latent vectors provided by inverting natural images, alternatives to SLERP have recently been proposed (Samuel et al. 2023 Zheng al. 2024). However, these procedures are costly, have parameters tune, and like SLERP - are challenging to generalise to other types of manipulations beyond interpolation. such constructing the subspaces needed for latent space optimisation methods Rromer-Bombarelii
ct al. 2018), or adapting to more diverse latent distributions beyond Gaussian.
In this work, demonstrate that successful generation from latent vectors is strongly dependent on whether their broader statistical characteristics match those of samples - a stronger condition than just having e.g. likely norms. We show that effective manipulation of latent spaces can be achieved by following the simple guiding principle of adhering to the modelling assumptions of the generation process. Our primary contributions are as follows:
We show that even if latent leads to a valid object upon generation, it can fail to provide effective interpolation if it lacks the typical characteristics of samples from the latent distribution - as diagnosed by statistical distribution tests.
We introduce Latent Optimal Linear combinations (LOL) an easy-to-implement method for ensuring that interpolation intermediates continue to match the latent distribution.
We show that LOL a closed-form transform that makes no assumptions about the network structure or data modality constitutes a Monge optimal transport map for linear combinations to a general class latent distributions.
We demonstrate that LOL can be applied to define general linear combinations beyond interpolations, including centroids (Figure 2 and, in particular, to define meaningful low- dimensional latent representations (Figure
goal achieved previously with significant
expense and only for specific architectures. Implementation and examples: https//ghhh..ccommdideeeeeeeeerh
Figure I: Low-dimensional latent subspaces. A 5 5-dimensional subspace from the flow matching model Stable Diffiusion 3 (Esser et al. 2024) extracted using LOL (left) from the latents corresponding to images Z1...., Tz. The left plot show generations from uniform grid points across an axis-aligned slice of the subspace coordinate system, centered around the coordinate for æ1. Each coordinate in the subspace correspond to a linear combination of latents, which define basis vectors. The right plot shows the corresponding subspace without the proposed LOL transformation. See Figure 6 Figure] and Section Glin the appendix for additional examples.
BACKGROUND
2.1 GENERATIVE MODELLING WITH LATENT VARIABLES
The methodology in this paper applies to any generative model that transforms samples from a known distribution in order to model another distribution - typically, complicated distribution of high-
2

---

Published as a conference paper at ICLR 2025
Euclidean
Euelideam
Euclidean
Figure 2: Centroid determination. Generation using Stable Diffusion 2.1 (Rombach et al. 2022) from the centroid of the latents corresponding images æ1, x2. 23 using different methods. Note that our proposed method removes several artifacts, such as unrealistic headlights and chassi texture.
dimensional objects. For clarity exposition, we focus on popular Diffuusion and Flow Matching models using Gaussian latents, and then extend general latent distributions in the appendix.
Diffusion models learn a process that reverses the effect of gradually adding noise to training data (Ho al. 2020). The noise level is governed by schedule designed so that the noised samples at the final time index T follow the latent distribution (T) p(æ). To sample from the diffusion model, one starts by drawing sample from the latent distribution before iteratively evaluating the reverse process with learnt parameters 0, denoising step-by-step generating a sequence {x(T), (T-1)) x (0))
(t-11) ~ pe (t ()
finally producing generated object x (0)) Diffusion has also been extended to the continuous time setting by [oong et al. (2020b), where the diffusion is expressed as a stochastic differential equation. Note that, by using the Denoising Diffusion Implicit Model (DDIM) (Song al., (020a) or the probability flow formulation in Song et al. (2020b), the generation process can be made deterministic, ie. the latent representation x! (T) p(x) completely specifies the generated object (0))
Flow Matching Lipman al. (2022) is an efficient approach to train Continuous Normalising Flows (CNF))Chen et al. (0018) an alternative class of generative models that builds complicated distribu- tions from simple latent distributions using differential equations. CNFs model the evolution of data points over continuous time using an ordinary differential equation (ODE) parameterized by a neural network, providing deterministic relationship between latent and object space. Mathematically. the transformation of latent sample (T) p(x) at time t T to (t)) time is governed by f(x("), t; 0), where f(; 0) is neural network with parameters 0. Flow matching allows the transformation dynamics of the CNF to be learnt without needing to simulate the entire forward process during training, which improves stability and scalability.
Note that both diffusion and flow matching models can generate in deterministic manner, where the realisation of the latent variable completely specifies the generated data object, e.g. the image. Moreover, by running their deterministic generation formulation reverse, one can obtain the corresponding latent representation associated with known object, referred to "inversion".. In this paper we focus on the effect of properties of the latent vectors, and their manipulation, on the generation process. For simplicity of notation, henceforth we drop the time index and refer to the latent variable x as σ, and use subscript indexing to denote realisations of the latent variable.
2.2 INTERPOLATION OF LATENTS
Linear Interpolation. The most common latent space manipulation is interpolation, where we are given two latent vectors x; and I2, referred as seed latents (or seeds), and obtain intermediate latent vectors. The simplest approach is to interpolate linearly between the seeds to get intermediates
y = w + (1 w)x2 for w E [0,
Spherical Interpolation. However, as discussed (White 2016) in the context of Variational
Autoencoders (Kingma 2013) and Generative Adversarial Networks (Goodfellow et al. 20144) linear interpolation yields intermediates with unlikely norms for Gaussian samples and results in highly implausible generated objects (e.g. all-green images). Consequently, it is common instead use
3

---

Published as a conference paper at ICLR 2025
spherical interpolation (SLERP) (Shoemake 1985).
which instead builds interpolants via
sin wl VSLERP sin 0 21 sin (1 [l1-w) w) for E [0, 1] cos 0
{x) v2)
sin 0
a x2
to maintain similar norms for the intermediates as the endpoints. SLERP has become popular and is
the standard method for interpolation also in the more recent models (Song et al. 2020a b).
Norm-awar wware Optimisation. Motivated by poor performance of SLERP when interpolating between the latent vectors corresponding inverted natural images (rather than using those sampled directly from the model's 's latent distribution), "Zheng et al. (2024) propose to add additional Gaussian noise to interpolants via the inversion procedure to control trade-off between generation quality and adherence to the seed images (i.e. the images corresponding to the latents at the interpolation end- points). Alternatively, (Samuel et al. (2023) advocate for Norm-Aware Optimisation (NAO) which uses back-propagation to identify interpolation paths 7 [0, 1] + Rd that solve
inf - log (+)())) s.t. 7()) = 11, 7(1) = 222-
f
where log P. RR + R is the log likelihood of the squared norm under its sampling distribution (D) for unit Gaussian samples. NAO can also calculate centroids of K latents by simultaneous optimisation of paths between the centroid and each respective latent.
All the proposed interpolation methods above aim to make the intermediates adhere statistics of Gaussian samples, however, require significant computation, have hyperparameters to tune, or lack theoretical guarantees. Moreover - in common with spherical interpolation - they are difficult to generalise beyond Gaussian latent variables to more general expressions of the latents beyond interpolation, such as building subspaces based on their span as required to build expressive low-dimensional representations.
In this work we will propose a simple technique that guarantees that interpolations follow the latent distribution if the original (seed) latents do, applies to general class of latent distributions, and enables us go beyond interpolation. Moreover, we will address how to assess the distributional assumption of the seed latents, to simplify the diagnosis of errors at the source.
SD2.1 (diffusion)
SD3 (flow matching)
2 Afja,
zrgmax p(] ID
argmax -
Nic.:
AlM, Xi)
argmax p(] ID
argmax p01 IB
Ai X0
-50000 -22000
200155 -20660)
Figure 3: Likelihood and norm insufficient. Column (1) of each panel shows image generated using a random sample from the associated Gaussian latent distribution for the diffusion model Rombach al. (2022) (left side) and the flow matching model of Esser al. (2024) (right side). Columns (2) and (3) both show images generated from latents with the most likely norm according to their respective latent distribution. Columns (2) use the same Gaussian samples as in columns (1) but rescaled to have this norm, also yielding realistic images. Meanwhile, columns (3) show the failed generation from constant vectors 87 scaled to have the most likely norm according to the latent distribution but lacking other characteristics (e.g. not having all-equal values) that the network was trained to expect, even though likelihood, A/ (sT:; μr, Σ) typical of real samples. Moreover, the distribution mode μ., which also lacks needed characteristics. has vastly higher log likelihood than any realistic sample; -33875 and 1135003 for the two models, respectively. See Table 2 for an example of failed generation using the mode μ.
3 ASSESSING VALIDITY OF LATENTS VIA DISTRIBUTION TESTING
Before introducing our proposed method, we first explore evidence for our central hypothesis that tackles a misconception underpinning current interpolation methods for Gaussian latents:
Having a latent vector with a norm that is likely for a sample from the latent distribution (i, Σ) not sufficient condition for plausible sample generation. Rather, plausible generation requires
4

---

Published as a conference paper at ICLR 2025
latent vector with characteristics that match those of a sample from (p, Σ) more generally (as evaluated by normality tests), with likely norm being only one such characteristic.
The characteristics of the random variable may be described by collection of statistics, e.g. its mean, variance or norm. Above, we hypothesise that a specified latent æ. has an extremely unlikely value for characteristic for which the network has come rely, then implausible generation will follow. A norm is unlikely it has a low likelihood under its sampling distribution; for unit Gaussians is (D), see Section While the norm has been shown often be important statistic (Samuel et al. 2023), our Figure 3 illustrates on a popular diffusion model (Rombach et al., 2022) and flow matching model (Esser al.) (024) that latent with a likely norm even the most likely can still induce failure in the generative process if other evidently critical characteristics are unmet.
Invenioon budget
Original
stpps
sueps
64
100
200
500
999 sleps
Image
Rec. arro::
D.340
0.339
0.230
0.0539
0.0295
0.0225 -2
0.0186
0.0199
0.0178 1.6e
p-valoe
58
-1
2.3e
7.de -3
6.4e
-1
1.4e
1.0
04-
Invention
10h Pvela
1200
-300
N ripected
300
50
10a
10
10P
102
Inversion (isteps)
Figure 4: Normality testing of latent vectors obtained from inversion. We show the LPIPS (Zhang
al. 2018) reconstruction errors (second row, red) of 200 inverted randomly selected images across 50 random classes from ImageNet (Deng et al. 2009), the p-values of their inversions (third row), and rejection rates (bottom row) of the Aalinoooro--Sminnne normality test applied to the corresponding latent obtained from inversion under various step budgets. We use the diffusion model of [Rombach et al. (2022), always using its maximum number of steps (999) for generation, and denote the 10th, 50th and 90th percentiles with black lines. The first row shows image reconstructions using its inversion at each budget, highlighted in red when the latent was rejected (p < 1e 1e-3), with the interpretation that the characteristics of the latent were unlikely for real sample according to the KS test. We note the strong correlation between inversion budgets providing low reconstruction realistic taking values likely to occur
errors and those for which the p-values of the latents by chance for real samples. However, as we will see in Figure [ there are still many latents with low reconstruction enror yet extremely low p-value, and this often severely affects the quality of its interpolants.
In general, neural networks have many degrees of freedom, and so difficult to determine and list the input characteristics that specific model relies on. Therefore, rather than trying to determine all necessary statistics for each model, we propose instead to rely on standard methods for distribution testing, classic area of statistics (Razali et al.) 2011 Kolmogorov 1933 Shapiro Wilk, 1965) - here exploring the (null) hypothesis that a latent vector I. E ER" is drawn from (H, E). Popular normality tests consider broad statistics associated with normality. For example, the Kolloggoov-SSmingg Smirnov (Kolmogorov (933) test considers the distribution of the largest absolute difference between the empirical and the theoretical cumulative density function across all values.
5

---

Published as a conference paper at ICLR 2025
Seeds
Interpolations
High recomstructoo error
Low onconsiruction urror but low piva
70
Seeds:
Low reconsprction orar and high pvallc
low reconstruction value
08
60
Seads:
but low
Proportin
3.0
Visual quality
Visual quality
Vicual quality
Vistal quality
Visual quality
Visual quality
Figure 5: Lack of normality is linked to failure of interpolants The left and middle panels shows LPIPS (Zhang al., 2018) reconstruction error (after 999 generation steps) and the Kolmogorov- Smirnov p-value for all inversions presented in Figure 4 split into two plots due to the vast dynamic range of p-values. Although latents with high (realistic) p-values tend to have low reconstruction errors, there are many latents with low reconstruction errors that also have low p-values. The right panel shows Q-Align visual quality scores (Wu et 2023) for spherical interpolants between pairs of inversions selected from matching ImageNet (Deng et al. 2009) image classes, demonstrating that choosing seed latents with both low reconstruction error (< 0.05) and high p-values (> 1e-9) allows us avoid low-quality interpolants that would arise when choosing seeds by reconstruction error alone. For reference, we include examples of interpolants at each visual quality level.
3.1 THE SAMPLE CHARACTERISTICS OF SEED LATENTS AFFECT THEIR INTERPOLATIONS
An ability to identify latents that lack the necessary characteristics for good quality generation critical when manipulating latents that are not acquired by explicitly sampling from the latent distribution. Indeed, we show below, using such deficient latents seeds for interpolation can severely affect the quality of interpolants, for example when interpolating between inverted natural images (the most common way to obtaín seed latent, see Section 2 As inversion is subject to a finite computation budget (i.e. finite inversion steps), numerical imprecision, and because the particular data instance may not be drawn from the distribution of the model's 's training data, the resulting latent vector æ, may not correspond to a sample from the latent distribution. We will now show that distribution testing is helpful for identifying such deficient latents, providing opportunity to apply corrections or review the examples or inversion setup, before performing interpolation.
Figure demonstrates that inversions having realistic p-values strongly correlated with them reproducing their original image with good quality. The p-value indicates the probability of observing the latent vector by chance given that it is drawn from the latent distribution A(p, E). If this value is extremely low that is strong indicator that the latent lacks characteristics expected of samples from (1,, E). In the figure we see that low inversion budgets (with high reconstruction errors) most v-vuuu are 1e 50 or lower, and then reach values you expect to see by chance for real samples (around le 3) at budgets where the reconstruction errors tend to be small. However, we will show that the p- valal provides additional information about the quality of the latents than provided by reconstruction error alone.
We now examine how the p-values of two inversions relate to our ability to interpolate them with good quality. Figure that although latents with realistic p-values tend to have low reconstruction errors, there are many latents with low reconstruction errors that have low p-values. Therefore, just because the process of going from object to latent and back reproduces the object, this does not necessarily mean that the resulting latent is well characterised as a sample from A (1, E). The lack of such characteristics can be inherited when manipulating the latent; in Appendix [ we show that interpolants typically inherit low p-values from their seed latents. Figure also demonstrates that if

---

Published as a conference paper at ICLR 2025
we beyond requiring inversions to reconstruct their original objects also require their p-values to be realistic - at levels expected of real samples - we are able to avoid many low quality interpolants downstream, by not using the rejected latents as seeds. In other words, the normality test helps us discover when the latents acquired from inversion lack characteristics of real samples, as this lack prevents them from being reliably used as seeds. Distribution testing the latents can be helpful for debugging the inversion setup which typically involves solving an ODE - and assessing the settings used. In Appendix B we investigate the effectiveness of range of other classic normality tests and tests based on the likelihood (xx.; je, Σ) as well as the likelihood of the norm statistic. We assess these methods across a suite of test cases including vectors with unlikely characteristics, and for which we know, through failed generation, violate the assumptions of the generative model.
4 LINEAR COMBINATIONS OF LATENTS
Now, equipped with the knowledge that matching broad characteristics of a sample from the latent distribution is critical, we propose a simple scheme for forming linear combinations of seed latents which we now will assume are sampled from the latent distribution - to maintain their sample characteristics. We will focus on Gaussian latents in this section, and present an extension to general latent distributions in Appendix A In this section we change the notation slightly, where a seed latent 2y E RD rather than being a realisation of a random variable - a vector of known values- - is a random variable following the latent distribution (here, ((,, Σ)). We assume access to K such seed latent variables { and attempt to form new variables following the same distribution.
Let y be a linear combination of the K Gaussian latent variables a NN, E)
WK = w X,
where Wik E R. w = [w1, W2...., W wx] and X = [21, 22,... xk]. Then we have that y is also a Gaussian random variable, with mean and covariance
K
K
y N(au,
Wk
8
Wkk
k=1
In other words, y is only distributed as N(H, Σ) in the specific case where (a) au f and (b) B = Σ- - an observation which we now use to explain the empirically observed behaviour of existing interpolation methods. Firstly, for linear interpolation, where w [u. w), v E [0, 1). (b) holds only for the endpoints a = {0, 1}. and so leads to implausible generations for interpolants (as we demonstrate empirically in Figure [I]. In contrast, in the popular case of high-dimensional unit
Gaussian latent vectors, spherical interpolants have 3 ~ 1, Ve E [0, 1]. as proven in Appendix C and (a) is met as a0 0, which is consistent with plausible interpolations (see Figure I1.
In this work, we instead propose transforming linear combinations such that a 3 = 1, for any w E K thus exactly meeting the criteria for any linear combination and any choice of μu and Σ. We define a transformed random variable z to use as the latent instead of the linear combination y
==
(x) = Tam.g
wg
where T Je (3) == (1 (1- ) 3/ for which it holds that z (N, Σ) given latent variables I6 (H,, E). Here, Teo y a linear combination of (Hu, -a-varialles with weights w into A(p, Σ). In Appendix B we show that this transport map is Monge optimal, and extend it to general distributions in Appendix A The weights w, which via a and 33 together with the set of K seed latents specify the transformed linear combination z, depend on the operation, represented as particular linear combinations. Below are few examples of popular operations (linear combinations) to form y: these are used as above to, for the corresponding weights w, obtain following the distribution expected by the generative model.
v : RRD + R is the map that transforms samples from the distribution of
Interppolation: y w X, where X [I01, M2), w [Wi, 1 ww]] and wh [0, 1).
7

---

Published as a conference paper at ICLR 2025
Centroid Determination: y w'X, where X [x1................... xk|, |
Subspaces:
Suppose we wish build navigable subspace spanned by linear combinations of K latent variables. By performing the QR decomposition of X := [21. ux] E DR to produce semi-orthonormal matrix U RDxK (as the Q-matrix), we can then define a subspace projection of any new æ into the desired subspace via s(x: == U x = Uh RD. The weights w for a given point in the subspace s(x) are given by (x) = X I where X is the Moore-Penrose inverse of X. See the derivation of the weights and proof in Appendix D One can directly pick coordinates h E R²* compute the subspace projection y Uh, and then subsequently use the transport map (defined by Equation 3 to the latent space to obtain z. In Figure | we use grids IR K set h, used above together with basis (defined by U) from set of latents.
5 EXPERIMENTS
We now assess our proposed transformation scheme LOL experimentally, To verify that LOL matches or exceeds current methods for Gaussian latents for currently available operations - interpolation and centroid determination we perform qualitative and quantitative comparisons to their respective baselines. We then demonstrate new capabilities with several examples of low-dimensional subspaces on popular diffusion models and popular flow matching model.
5.1 INTERPOLATION AND CENTROID DETERMINATION
For the application of interpolation, we compare our proposed LOL to linear interpolation (LERP), spherical linear interpolation (SLERP), and Norm-Aware Optimization (NAO) Samuel al. 20233
In contrast to the all the other considered interpolation methods (including LOL) which only involve closed-form expres- sions, NAO requires numerical optimisa- tion. We closely follow the evaluation protocol in Samuel et al. (2023), basing the experiments on Stable Diffusion (SD) 2.1 (Rombach al.) 2022) and inversions of random images from 50 random classes from ImageNet (Deng et al. 2009), and assess visual quality and preservation of se- mantics using Fréchet Inception Distance (FID) (Heusel et al.] 2017) and class pre- diction accuracy, respectively. For the in- terpolation we (randomly without replace- ment) pair the 50 images per class into 25 pairs, forming 1250 image pairs in total.
Interpolation
Method
Accuracy
FID +
Time
LERP
3.92%
199
6e "s
SLERP
64.6%
42.6
NAO
62. 1 %
46.0
30s
LOL (ours)
67.4%
38.9
6e "s
Centroid determination
Method
Accuracy
FID 1
Time
Euclidean
0.286%
310
4c "s
Standardised Euclidean
44.6%
88.8
1e
Mode nomm Euclidean
44.6%
88.4
le 3s
NAO
44.0%
93.0
90s
LOL (ours)
46.3%
87.7
6e
Table 1: Quantitative comparisons of baselines.
For the centroid determination we compare to NAO, the Euclidean centroid E two transformations thereof; "standardised Fuclidean", where z is subsequently standardised to have mean zero and unit variance (as SD 2.1 assumes), and "mode norm Euclidean", where æ is rescaled for its norm to equal the maximum likelihood norm VD. For each class, we form 10 3-groups, 10 5 groups, 4 10-groups and 25-group, sampled without replacement per group, for a total of 1250 centroids per method. For more details on the experiment setup and settings, see Appendix F
=k=1 I& and
In Table we show that our method outperforms or maintains the performance of the baselines in terms of FID distance and accuracy, as calculated using a pre-trained classifier following the evaluation methodology Samuel et al. (2023). For an illustration of centroids and interpolations, see Figure and Figure [6] respectively. The evaluation time of an interpolation path and centroid, shown with one digit of precision, illustrate that the closed-form expressions are significantly faster than NAO. Surprisingly, NAO did not perform as well as spherical interpolation and several other baselines, despite using their implementation, which outperforming these methods in Samuel jet al. (2023). We note one discrepancy is that we report FID distances using (2048) high-level features, while their work they are using (64) low-level features, which in Seitzer (2020) is recommended
8

---

Published as a conference paper at ICLR 2025
against it does not necessarily correlate with visual quality. In the appendix include FID distances using all settings of features. We note that, in our setup, the baselines perform substantially including NAO in terms of FID distances using the 64 low-level feature setting (1.30 in our setup vs 6.78 in their setup), and class accuracy during interpolation (62% vs 52%). See Section H in the appendix for more details and ablations.
better than reported in (Samuel et al. (2023)
Figure 6: Low-dimensional subspaces. The latents Z1..... 0; (corresponding to images) are converted into basis vectors and used to define 5-dimensional subspace. The grids show generations from the flow matching model Stable Diffusion (Esser et al. 2024) over uniform grid points in the subspace coordinate system, where the left and right grids for the dimensions (1,2) and {3, 4), respectively, centered around the coordinate for æ. Each coordinate in the subspace correspond to a linear combination of the basis vectors, which through LOL all yield high-quality generations.
5.2 LOW-DIMENSIONAL SUBSPACES
-50
-00
Figure 7: Model-agnostic subspace definitions. The latents æ1, 22 and 33 which via the generative model correspond to 3D designs three-dimensional subspace. The left plot shows the generated designs corresponding to a ten-by-ten grid in a two-dimensional slice of the subspace, shown over a region. The right plot shows the wavedrag, evaluated by simulation in Open VSP (McDonald & Gloudemans) 2022) for the designs over 100-by-100 grid in the same region. with the respective design coordinates (from the left plot) shown as black dots. We trained the SLIDE (Lyu et al. (202)) diffusion model on ShapeNet (Chang al. 2015). LOI.. allows subspaces to be defined without any model-specific treatment.
converted into basis vectors and used to define
In Figure D and Figure we illustrate slices of a 5-dimensional subspace of the latent space of flow matching model, indexing high-dimensional images of sports and rocking chairs, respectively.

---

Published as a conference paper at ICLR 2025
The subspaces here are defined using five images (one per desired dimension), formed using our LOL method described in Section to transform the linear combinations of the corresponding projections. Our approach is dimensionality and model-agnostic which we illustrate in Figure 7 where the same procedure is used on a completely different model without adaptations - where we define three-dimensional subspace based on three designs in point cloud diffusion model with mesh reconstruction. We evaluate designs in this subspace using simple computational fluid dynamics simulation to illustrate that our approach allows objective functions be defined over the space, in turn allowing off fff-the-shelf optimisation methods to be applied to search for designs. In Figure 9 in the appendix we show corresponding grids to the sports cars in Figure without the proposed LOL transformation, which either leads to implausible images or the same image for each coordinate, depending on the model. In the appendix (Section Gb we also include more slices and examples; including subspaces using the diffusion model Stable Diffusion 2.1 (Rombach et al. 2022)..
6 RELATED WORK
Generative models with non-Gaussian priors. In Fadel et al. (2021) | and Davidson et al. (2018) 8). in the context of normalizing flows and VAEs, respectively, Dirichlet and von Mises-Fisher prior distributions are explored with the goal of improving interpolation performance. However, these methods require training the model with the new latent distribution, which is impractical and untested for the large pretrained models we consider.
Conditional Diffusion. An additional way to control the generation process of a diffusion model is to guide the generative process with additional information, such as text and labels, to produce outputs that meet specific requirements, such as where samples are guided towards a desired class (Dhariwal & Nichol 2021 Ho & Salimans] 2022). or to align outputs with text descriptions (Radford et al. 2021). Conditioning is complementary to latent space manipulation. For example, when making Figure we used conditioning (a prompt) to constrain the generation to sports cars, however, the variation of images fulfilling this constraint is encoded in the latent space.
Low-dimensional representations. We have shown that LOL can provide expressive low- dimensional representations of latent spaces of generative models. To the best of our knowledge, the most closely related line of work initiated Kwon et al. (2022), where it was shown that semantic edit directions can recovered from activations of a UNet (Ronneberger et al. 2015) denoiser network aarguage-Image Pre- Tre-Trainng) (Radford) iet al. 2021) model to define directions within the inner-most feature map of the UNet architecture, named h-space, they show that some semantic edits, such as adding glasses an image of a per- son, correspond to linear edits in the h-space. More recently, Haas et al. (2024) demonstrate that h-space edit directions can also be found through Principle Component Analysis, and Park et al.) (2023) propose a pullback metric that transfers edits in the h-space into the original latent space. However, while these three approaches demonstrate impressive editing capabilities on images, they all require modification to the generative process and are limited to diffusion models built with UNet architectures. In our work low-dimensional representations are instead formed in flosed-form as linear subspaces based a (free) choice of latent basis vectors. These basis vectors could be obtained through various methodologies, including via the pull-back metric in Park et al. (202)). which may be interesting to explore in future work.
during generation. Using a pretrained CLIP (Contrastive
7 CONCLUSION
In this paper we propose LOL, general and simple scheme to define linear combinations of latents that maintain prespecified latent distribution and demonstrate effectiveness for interpolation and defining subspaces within the latent spaces of generative models. Of course, these linear combinations only follow the desired distribution if the original (seed) latents do. Therefore, we propose the adoption of distribution tests to assess the validity of latents obtained from e.g. inversions. Still, the existing methods for distribution testing may not align perfectly with a given network's expectations e.g. many tests may be stricter than necessary for the network and it may be an interesting direction of future work to develop tailored methods, along with more extensive comparisons of existing tests in the context of generative models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank Samuel Willis and Daattavya Aggarwal (University of Cambridge) for their valuable contributions. Samuel contributed important ideas related to computational fluid dynamics simulation and the SLIDE model, and constructed the setup and plots used in Figure 7 Daattavya assisted in verifying proofs and offered comments that greatly improved the manuscript.
REFERENCES
Robert B Ash. Information theory. Courier Corporation, 2012.
Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 151.00151 2015.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Harald Cramér. On the composition of elementary errors: First paper: Mathematical deductions. Scandinavian Actuarial Journal, 1928(1):13-74, 1928.
Tim Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M Tomczak. Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. pp. 856-865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Samuel G Fadel, Sebastian Mair, Ricardo da S. Torres, and Ulf Brefeld. Principled interpolation in normalizing flows. In Machine Learning and Knowledge Discovery Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part 11 21, pp. 131. Springer, 2021.
Rafael Gómeez-Bombarelli Jennifer N Wei, David Duvenaud, José Miguel Hernández-LLoatto Benjamín Sánchez- Lengeling, Dennis Sheberla, Jorge caalllrra-pprragiiree. Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
René Haas, Inbar Huberman- Sppirellll,,iggegllaa, Rotem Mulayoff, Stella Graßhof, Sami S Brandt, and Tomer Michaeli. Discovering interpretable directions in the semantic latent space of diffusion models. In 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1-9. IEEE, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 1rX1v:2207.12598 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilisticc models. Advances in neural information processing systems, 15:6840-6855 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
An Kolmogorov. Sulla determinazione empirica di una legge didistribuzione. Giorn Dell' inst Ital Degli Ait, 4:89-91, 1933.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0976,, 2020.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.1090 2022.
Richard Larsen and Morris L Marx. An introduction to mathematical statistics. Prentice Hall Hoboken, NJ, 2005.
Yaron Lipman, Ricky TQ Chen, Heli Gen-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Zhaoyang Lyu, Jinyi Wang, Yuwei An, Ya Zhang. Dahua Lin, and Bo Dai. Controllable mesh genera- tion through sparse latent point diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 271-280, 2023.
Robert A McDonald and James R Gloudemans. Open vehicle sketch pad: An open source parametric geometry and analysis tool for conceptual aircraft design. In AIAA SciTech 2022 Forum, pp. 0004, 2022.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs deep generative models using typicality. arXiv preprint arXiv: 1906.02990 2019.
Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129-2411 2023.
Karl Pearson. X. on the criterion that given system of deviations from the probable in the case of correlated system of variables such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science,
1900.
Gabriel Peyré and Marco Cuturi. Computational optimal transport, 2020. URL https://axxiv Drg/abs//8033 00567
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Nomadiah Mohd Razali, Yap Bee Wah, al. Power comparisons of shapiro-wilk, kolmogorov- smirnov, lillicfors and anderson anderson-darling tests. Journal of statistical modeling and analytics. 2(1): 21-33, 2011.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234-241. Springer, 2015.
Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm nomm-guied latent space exploration for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2023.
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://ggituuu..oommssettttt pytorch- fid, August 2020. Version 0.3.0.
Samuel Sanford Shapiro and Martin B Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4):591-111 1965.
Ken Shoemake. Animating rotation with quaternion eurves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Michel Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l'Institut des Hautes Études Scientifiques 81:73 205, 1995.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pp. 459-479. Springer, 2022.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2009.
Tom White. Sampling generative networks. arXiv preprint arXiv: 2016.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17000, 2023.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
PengFei Zheng, Yonggang Zhang, Zhen Fang, Tongliang Liu, Defu Lian, and Bo Han. Noisediffusion: Correcting noise for image interpolation with diffusion models beyond spherical linear interpolation. arXiv preprint 2at.:2240000888440 2024.
A OPTIMAL TRANSPORT MAP FOR GENERAL LATENT LINEAR COMBINATIONS WITH INDEPENDENT ELEMENTS
In Section we introduced LOL for Gaussian latent variables, the most commonly used latent distribution modern generative models. We now extend LOL to general latent distributions with independent real-valued elements, provided the cumulative distribution function (CDF) and its inverse known for each respective element.
Let p(x) be distribution with independent components across elements, such that p(æ) = Ih (1))) where pa is the distribution of element (d)) R with CDF &h and inverse CDF 07-
13

---

Published as a conference paper at ICLR 2025
Given linear combination weights {w> Ie wy R, and seeds { 2y p. obtain p (d)
via the transport map T(w) = (T((u) ld=1 defined per latent element 2d4 as:
2(4) = T(m) (1=x = (
( (1)) (d) = =To (E w
& 1 (f)(z1))
(4)
where P. the CDF of the standard Gaussian, and Two IR + IR the transport map for standard Gaussians as defined in Equation [3] We will now show that the proposed map leads to the target distribution and then show that this map is Monge optimal, leveraging results from Section
Lemma 1. Let z be a transformed variable defined through the transport map T(w) applied to linear combination of independent latent variables D& ~ p(æ)), such that
2(4) (1 (1=(00) (9)
(d))))
Wk Ux
where
c (4) - (
Then, z ~ p(æ)), meaning that the transformed variable follows the target latent distribution.
Proof. We prove that z p(x) by verifying that each element 2(d) follows the corresponding marginal distribution pd-
Pa, its cumulative distribution function (CDF) is given by ()(=(()) Defining the standard normal equivalent c क. (1 (d) )), we know that cl No.. 1) because the
Since a/(()
transformation via -- preserves uniformity. Now, consider the weighted sum:
K
(4)
(d)
(d) Since the & are independent standard normal variables, the resulting sum follows:
(d) N(a 0, 3. (0,
where a Ek= wh and B Σ W
Applying the optimal Gaussian transport map Tw. (as proven in Section B to e e(d) we obtain:
(d)
Tw (w)) = vB
Since (4) ~ (N, 8), it follows that:
1).
Thus, applying the standard normal CDF, we get:
Φ (T (TT ~ (0,
which is uniform distribution.
Finally, applying the inverse CDF & / of the target distribution ps results in:
(d) $ (T)))(((d)))) ~ Pd-
Since this holds for each d, we conclude that z ~ p(æ). proving that the transport map correctly transforms the linear combination into the desired latent distribution.
14

---

Published as a conference paper at ICLR 2025
Lemma 2. The transport map [w)) defined element-wise as
K whEk
z(d)
7 (1)
(1
(d))
})
=b7 (
(To.
(d)
)))
where
(4) - ( c
the Monge optimal transport map, minimizing the quadratic Wasserstein distance W₂ between the source distribution of the weighted sum of independent elements and the target latent distribution
Proof. We want to show that T(w minimizes the quadratic Wasserstein distance W2 between the source distribution (the weighted sum of independent seeds) and the target distribution.
Step 1: General Theorem for One-Dimensional Optimal Transport For two probability
sures P and Q on R with cumulative distribution functions (CDFs) Fp and Fo, respectively, the optimal transport map in the W2 sense (which minimizes the expected squared Euclidean distance) is given by the monotone increasing function:
T T' F Fp.
This result follows from classical optimal transport theory for univariate distributions (see Remark 2.30 in Peyré Cuturi 2020)). )
Step 2: Applying to Our Case For each element d, the source distribution is given by the weighted sum:
K
(d)
Each Eg (d)
is obtained by transforming the original variable T (d)
using:
c (d) = ($a(
Since the c (d) independent standard normal variables, their weighted sum follows:
K
(d) ~ N),,3), where B
=
w
1=1
To map this distribution to the standard normal (0, 1), the optimal transport map for Gaussians (derived in Section is:
Tw ((())( (d)
vB
Applying the standard normal CDF Φ to this transformation ensures that:
T (Tw ~ U(,
By the general optimal transport theorem stated Step 1, the optimal transport map to the target distribution Pa is then:
2(d) 1 (T (T
15

---

Published as a conference paper at ICLR 2025
Since each element d is mapped independently using the element- Monge optimal transport map, the full transformation (1c) is optimal in the Monge sense, minimizing the quadratic Wasserstein distance W2.
Thus, T(w is the Monge optimal transport map from the distribution of the weighted sum of independent elements the target latent distribution p, completing the proof.
B OPTIMAL TRANSPORT MAP FOR GAUSSIAN LATENT LINEAR COMBINATIONS
In Section we introduced LOI., which binations for example interpolations or subspace projections - to the latent distribution M(,, Σ) via transport map Tw. defined per set of linear combination weights We will now derive the distribution of (uncorrected) linear combination y - which we show does match the latent distribution followed by the distribution of the transformed variable z, which we show does. We will then show that the proposed transport map Tw, is Monge optimal.
transform random variables arising from linear
[Wi...., WK ) wx E R.
Lemma 3. Let y be linear combination of K i...d random variables an. NMe, E), where is defined as
WRFE w X,
y
with wk E R. y.yeR 2& E w [11, U2.... wk). K E Nso and X = [21, Tk] Then Gaussian random variable with the distribution
xk].
y ~ Nop,, BE),
where
and
B ==
k=1
Proof. Given (n, E), we know that each has mean and covariance matrix Σ.
Define y w T
Σ K k= W
First, we calculate the mean of y:
K wy
K
E Elyl = E
Wg
EE
[c
Wkff
Σ k=1
W6k
/ ap.
k=1
Next, we calculate the covariance of
K Σ
Cov(y) Cov Σ wkar
w Cov
A=1
since the are i.i.d. and thus (o)( xx()) 0 for i A.
Given that mg NH,,, E), we have Cov(xx)) Σ. Therefore,
K
K
Cov(y)
2 k
Σ
W
Σ = =
k=1
k=
Hence, ~ N(ou,, with a = Ek We and 3 = Σk w
16

---

Published as a conference paper at ICLR 2025
Lemma 4. Let be defined as
Te)) (1 with y ~ Nou,, 32). Then 2 Nu, E)
(z)
v3
k
vB
y
a
Wk
Wi
Proof. Given y ~ (ayu, BΣ), we need to show that NM,, 2))
First, we calculate the mean of z:
[[z] E
By] = a + BEIy) = (1-
p +
ap p.
vB
VB
where
a = 1
VB and B=
vB
Next, we calculate the covariance of z:
Cov(z) = Cov (a + By) = BC
3 E.
Since z has mcan f and covariance Σ, we conclude that
2 ~~ NH, E).
Lemma 5. The transport map Tw.,p defined by
α (1-
y /B
Tw (w) = =1
VB
is the Monge optimal transport map, minimizing the quadratic Wasserstein distance W₂ between Nop,,333 and (p,, E).
Proof. In optimal transport theory, the Monge problem seeks transport map T R R that pushes forward the source distribution A (Jty, Ey) to the target distribution N(1z,, Σz) while minimizing the expected squared Fuclidean distance:
W E.)) inf (y) y11l
TEM
The optimal transport map for Gaussian distributions is given by the affine transformation:
T(y) = uz + A(y My),
where A is defined as
A = Σ
See Remark 2.31 in Peyré & Cuturi (202)).
Setting the parameters
Hy = axu.
Σ, = BΣ,
(5) (6)
f4 μ.
Σ, E,
17

---

Published as a conference paper at ICLR 2025
compute A:
A = (0E) (1)))
1/2
I.
vB
Thus, the optimal transport map simplifies to
y
2 :: T = v3 yy aq) = (1
(8)
vB
vB
Since this matches exactly with Tur we conclude that Tan,j a the Monge optimal transport map minimising the quadratic Wasserstein distance.
C SPHERICAL INTERPOLANTS OF HIGH-DIMENSIONAL UNIT GAUSSIAN RANDOM VARIABLES ARE APPROXIMATELY UNIT GAUSSIAN
Spherical linear interpolation (SLERP)
Shhoemake 1985 is defined
y W1 W2
(9)
where
sin 0
W/
(10)
sin
w E (0, 1] and v2 1 vj and cos
(.),
where cos 0 is typically referred to the cosine
similarity of œ1 and æ2. As such, using Equation we obtain
sin 0
sin 020
and 3 sin" 0.
sin 020 (11)
wk
sin
sin
sin
sin 2
As discussed in Section 4 for a linear combination y be random variable following distribution M(p,, E), given that æ; and Ø2 do, must be true that au = l and 3Σ Σ.
common case using unit Gaussian latents (as in e.g. the models Esser et al. (2024) and Rombach et al. (2022) used in this paper), i.e. where = 0, Σ J. In this case trivially follows that au l since a0 O. We will now show that 3 xz in this special (i.c. unit Gaussian) case.
Lemma 6. Let
3 sin? v0
sin (1 (1-r)0
+
sin 2
sin 0
where cos
(z) and 1 D2 N(0, I). Then 3 ze 1 for large D. Ve [0, 1]. 21
22
Proof. Since 11 x2 standard normal random variables. The inner product (æ11 222) is given by:
NO. I), each component æıj and ø2 for j 1,... D are independent
D
(x) x2))
T1j32j.
The product 22 follows distribution known as the standard normal product distribution. For large D, the sum of these products is approximately normal due to the Central Limit Theorem (CLT), with:
(x), v2) NO, D)-
18

---

Published as a conference paper at ICLR 2025
Next, consider the norms ||æ1|| and || Each ||,,| with degrees of freedom. For large D, by the central limit theorem, ||x;|22 (D, 2D), and therefore [e;|] is approximately vD.
|= - zi is chi-squared random variable
Thus, for large D,
(x) 222
D)
N(O,D)
N 0, 1 )
cos cos(0)
12
VD-
D
D
Thus, 0 R 7/2, which implies sin(0) ~ 1. Therefore:
sin (v0)
sin din v)0) sin 0
sin) + sin v)0).
sin 0
Using the identity sin (a) (a) sin (b) cos b),
3 - cos (v0 - e)0 = = co)
Given v [0,1] and 9 n/2 for large D, the argument of remains small, leading cos(-) 0. Hence,
B Rd
Therefore, for large D, follows that
1.
Empirical verification To confirm the effectiveness of the approximations used above for values of D which are typical for popular generative models that unit Gaussian latents, estimate the confidence interval of 3 using 10k samples of æ and Ø2, respectively, where æ; ~ (O,, I). For Stable Diffusion 3 (Bsser al. 2024), flow matching model with D 147456, the estimated 99% confidence interval of 3 is [0.9934, 1.0067] for e 0.5 where the eπror is largest. For Stable Diffusion 2.1 (Rombach of ( is [0.9868, 1.014].
2022). a diffusion model with D 36864, the corresponding confidence interval
D LINEAR COMBINATION WEIGHTS OF SUBSPACE PROJECTIONS
4 we introduced LOL, which we use to transform latent variables arising from linear combinations such that they maintain the the latent distribution. This transformation depends on the weights which specify the linear combination. We will now derive the linear combination weights for subspace projections.
In Section
Lemma 7. Lei U DDX a semi-orthnormal matrix. For a given point x E D..,, the subspace projection is s(x) = UUT x. The weights E R&K such that s(x) is linear combination of 3. B22 ........ KKK (columns of [m1, 22, xx E RDDX can be expressed as ur A45(x), where A, is the Peoorse inverse of A.
Proof. The subspace projection s(x) of R is defined as:
s = UU x.
We aim to express s(x) as a linear combination of the columns of = == (2.,, ØK| E DxK That is, we seek RR such that:
s s(x) Aww
By definition, the Moore-Penrose inverse A, of A satisfies the following properties:
1. A A
2. A, A
19

---

Published as a conference paper at ICLR 2025
3. (AA,)* AA,
4. (A₁ = A,A
Since s(x) is in the subspace spanned by the columns of A, there exists such that:
<(x) Am.
Consider w' E R² constructed using the Moore-Penreee inverse A₁:
w A+)(z).
We now verify that this w' satisfies the required equation. Substituting back Aw' = (z)))
and using the property of the Moore-Penross inverse AA, A A, we get:
AA,s(z) s(x).
Thus:
Aw' = s(x),
which shows that w/ sax((z)) is indeed the correct expression for the weights.
From uniqueness of w for a given set columns 21, I...... zk (see Lemma we have proven that the weights w for given point in the subspace s(x) are given by: A+s(x:)
Lemma 8. The weights w E RRKK such that s(x) is a linear combination of Iı, Z2..... 2K (columns of A [1, $2......... xK] E DxX) are unique.
Proof. Suppose there exist two different weight vectors and wy such that both satisfy the equation:
(ax) = Aw, Auz-
Then, subtracting these two equations gives:
Aw Aw = 0.
This simplifies to:
A(w, W2) = 0.
Let W2. Then:
Au = 0.
Since E K this equation implies that lies in the null space of A. However, the assumption that A has full column rank (since is used to represent a linear combination for s(x)) implies that A has no non-zero vector in its null space, i.e., Au 0 only when = 0. Therefore:
v
wi W2.
This shows that the weights w are unique, and there cannot be two distinct sets of weights w; and W2 that satisfy the equation s(x) Aw.
Hence, we conclude that the weights w such that s(x) is a linear combination of Φ; Ø2. EK are unique.
20

---

Published as a conference paper at ICLR 2025
E NORMALITY TEST COMPARISON
We will now investigate the effectiveness of range of classic normality tests and tests based on the likelihood ((x.; f4, Σ) as well as the likelihood of the norm statistic. We assess these methods across suite of test cases including for vectors with unlikely characteristics and which we know, through failed generation, violate the assumptions of the model. In order to improve statistical power and detect even small deviations from normality of single instance F. E IR we transform the single high-dimensional latent into large collection of (D) i.i.d. unit Gaussian samples {<] and test the hypothesis 61..... Ed ~ (0,1)) where c = Σ (a p.) [ei..... ea]. Note that rejecting the hypothesis E,, EI,..., E. (N(0, 1) corresponds to rejecting the hypothesis that x. ~ (H, )).
We compare popular alternatives for normality testing that assume a known mean and variance and that are applicable to very large sample sizes, including Kolmogorov-Smiroov (Kolmogorov, 1933) Shapiro- Wilk (Shapiro Wilk [965).. Chi-square (Pearson 1900), and Cramér-von Mises (Cramér 1928). We assess these methods on suite of test cases of vectors which are extremely unlikely under the model's latent distribution and which we know through failed generation violates the characteristic assumptions of the diffusion model, sample does follow the latent distribution, V, ~ (H, E). We report the [0.1%, 99. -coniicineeeeeceee interval of the p-value produced by each respective method Te random samples in the stochastic test cases. We also include the likelihood N (xx: Js, E) and the likelihood of the norm statistic.
well as assess the "positive case" where the
The results reported in Table 2 We find that Kolmogorov-Smimoo and Cramér-von Mises both succeed in reporting a lower 99.9th percentile p-value for each tested failure case than the 0.1th percentile assigned to real samples. Kolmogorov-Smirno did so with the largest gap, reporting extremely low p-values for all failure cases while reporting calibrated p-values for the positive case. Shapiro- Wilk, which in the literature often is regarded as one of the most powerful tests (Razali et al. (01T). did not work well in our context in contrast to the other normality testing methods, as produced high p-values also for several of the failure cases. An explanation may be that in our context we have sample sizes of tens of thousands (same as the dimensionality of the latent D), while comparisons in the literature typically focuses on smaller sample sizes, such as up to hundreds or a few thousand (Razali et al. 2011). The Chi-square test is test for discrete distributions. However, it commonly used on binned data (Larsen & Marx 2005) to approximate a continuous distribution as discrete. We do this, using histograms of 30 bins. This test successfully produced very low p-values for each failure case, but also did so for many valid samples, yielding a 0.1% of O. The likelihood N(x.; Ja. Σ) assigns high likelihoods to vectors æ, the mode irrespective of its characteristics. We note that it fails to distinguish to failure cases from real samples, and in some cases assigns much higher likelihood to the failure cases than real data. The insufficiency of the likelihood to describe the feasibility of samples is a phenomenon present in cases of high-dimensionallty and low joint dependence between these dimensions; this due to the quickly shrinking concentration of points near the centre (or close to μr, where the likelihood is highest) as the dimensionality is increased (Talagrand] [995] Ash 2012 Nalisnick et al. 2019). The norm statistic we note successfully assigns low likelihoods to some failure cases, where the norm is atypical, but fails to do so in most of the tested cases.
In summary, we find that Kolmogorov-Smimoo and Cramér-von Mises both correctly rejects the failure cases (by assigning low p-values) while reporting calibrated p-values for real samples, with Kolmogoroy-Smirno doing with the lowest p-values for all failure cases.
F INTERPOLATION AND CENTROID DETERMINATION SETUP DETAILS
Baselines For the application of interpolation, we compare to linear interpolation (LERP), spherical linear interpolation (SLERP), and Norm-Aware Optimization (NAO) (Samuel al., 2023), a recent approach which considers the norm of the noise vectors. In contrast to the other approaches which only involve closed-form expressions, NAO involves a numerical optimization scheme based on a discretion of a line integral. For the application of centroid determination we compare to NAO, the Euclidean centroid ã= ΣK ised Euclidean", where æ is subsequently standardised to have mean zero and unit variance, and "mode norm Euclidean", where & is rescaled have the norm equal to the (square root of the) mode
Dk=1 Ik. and two transformations of the Euclidean centroid; "standard-
21

---

Published as a conference paper at ICLR 2025
Test CaMe:
2 - 0.
24
2: -.1
Sr. reselled F& rescaled
Methedivaloe
Ig. M J
1- -
io y [| (2)
lel
76
24
21
Staploo Wilk
12 4.0
Ch-equare
0.00
10.00
Cranse- Miness
0.0
1e-4
14-7 4:
44
22 6-
Kolmogoton Smirmo
Table 2: Normality testing of latents Reported is the [0.1%, 99.9%999999/.. confiideee interval of the p value produced by each respective normality testing method on lea random samples in the stochastic test cases. We also report the log likelihood of the vector (+)) under the latent distribution and the likelihood of the norm statistic. The diffusion model in Rombach et al. (2022) is used in this example. Green colour for the respective failure test case indicate the method assigning a lower p-value or likelihood to the failure case to least 99.9% of the failure samples than the lowest 0. 1% of real samples, to illustrate that it was successful in distinguishing the failure samples. Chi-square assigns low p-values many real samples as well, which we indicate in red. The images show the generated image from a random failure sample of the test case. In (failure) Test 1 in column I the latent is 0, the mode of the latent distribution for this model (0 I). In Test 2 it is the constant vector with the maximum likelihood norm of the norm sampling distribution. In Test 3 the constant vector is instead normalised to have its first two moments matching the latent's 's marginal distribution. n()) indicate that the vector(s) of the test case is normalised to have zero mean and unit variance. In Test 4 a linspace vector ,-,,...,1, 1i have been similarly normalised. In Test and the normal samples (from the left most column) have been transformed with scaling and bias, respectively. In Test 7 random selections of 1% of the dimensions [1, 11....... D] of the normal samples have been repeated (100 times) and subsequently normalised by nt()) In Test 8 and 9 a proportion of the dimensions of the normal samples have been multiplied by 5 (with indices selected randomly) and the whole vectors subsequently normalised by n(.), where the proportion is 5% and I 1,, respectively.
of x² (D), the chi-squared distribution with D degrees of freedom, which the maximum likelihood norm given that æ has been generated from unit Gaussian with D dimensions.
Evaluation sets We closely follow the evaluation protocol in Samuell al. (2023), where we base the experiments on Stable Diffusion 2 (Rombach et al. 2022) and inversions of random images from ImageNetIk (Deng et al. 2009). We (uniformly) randomly select 50 classes, each from which we randomly select 50 unique images, and find their corresponding latents through DDIM inversion (Song) jet al., (022a)) using the class name as prompt. We note that the DDIM inversion can be sensitive to the number of steps. Therefore, we made sure to use a sufficient number of steps for the inversion (we used 400 steps), which we then matched for the generation; see Figure 15 for an illustration of the importance of using sufficient number of steps. We used the guidance scale 1.0 (i.e. no guidance) for the inversion, which was then matched during generation. Note that using no guidance important both for accurate inversion as well as not introduce a factor (the dynamics of prompt guidance) which would be specific to the Stable Diffusion 2.1 model.
For the interpolation setup randomly (without replacement) pair the 50 images per class into 25 pairs, forming 1250 image pairs in total. In between the ends of each respective pair, each method then produce three interpolation points (and images). For the NAO method, which needs additional interpolation points to approximate the line integral, we used 11 interpolation points and selected three points from these at uniform (index) distance, similar to in Samuel al. (2023).
For the centroid determination setup we for each class form 10 3-groups, 10 5-groups, 4 10-groups and 1 25-group, sampled without replacement per group i.c. each method is to form 25 centroids per class total, for an overall total of 1250 centroids per method. Similarly to the interpolation setup, for NAO we used 11 interpolations points per path, which for their centroid determination method entails K paths per centroid.
But with replacement across groups, i.c. the groups are sampled independently from the same collection of images.
22

---

Published as a conference paper at ICLR 2025
-50
logie p-value of slerp Intarpolation
log p-olue of interpalction
-100
-150
-200
-250
-300
-10
-300
-250
-00
-150
-100
-50
-10
-8
22
logis minimum p-value
logso minimum p-value seeds
Figure 8: Low p-values are inherited by interpolants The x-axis show the lower Kolmogorov- Smirnov p-value of the two seed latents for each corresponding spherical interpolation in Figure [] and the y-axis show the p-value of the resulting interpolant. Due to the vast dynamic range of the p-values, the scatter plot is shown full on the left, and zoomed on the upper-right quadrant on the right. We note that when any of the two seed latents have small p-value this largely tend to be inherited by the interpolant. The Pearson correlation coefficient is 0.79 for the indicator acceptance (defined as 1 if the p-value is greater than 1e and 0 otherwise) and 0.66 for the log10 p-values.
Evaluation We, as in Samuel et al. 2023), assess the methods quantitatively based on visual quality and preservation of semantics using FID distances and class prediction accuracy, respectively.
The FID distances are computed using the pytorch-fid library (Seitzer 2020), using all evaluation im- ages produced per method for the interpolation and centroid determination respectively, to maximize the FID distance estimation accuracy.
For the classification accuracy, we used a pre-trained classifier, the Max ViT image classification model (Tu et al. (2022) as in (Samuel et al. (2023). which achieves a top- of 88.53% and 98.64% top-5 accuracy on the test-set ImageNet.
See results in Section
G ADDITIONAL. QUALITATIVE RESULTS
See Figure for demonstration of subspaces of latents without the LOL transformation introduced in Equation using a diffusion model (Rombach et al., (022) and flow matching model (Esser 2024), respectively. The setup is identical (with the same original latents) as in Figure 12 and Figure respectively, except without applying the proposed (LOL) transformation.
et al.
See Figure 10 for additional slices of the sports car subspace shown in Figure
See Figure and Figure for Stable Diffusion 2.1 (SD2.1) versions of the Stable Diffusion (SD3) examples in Figure and Figure O with an otherwise identical setup including the prompts. Just like the SD3 examples LOL defines working subspaces. However, as expected since SD2. is an older model than SD3, the visual quality of the generations is better using SD3.
See Figure for an interpolation example.
In the examples above the SD2.1 and SD3 models are provided with a text prompt during the generation. See Figure for an example where the original latents ((x,)) were obtained using DDIM inversion (from images) without prompt (and guidance scale 1.0, i.e. no guidance), allowing generation without a prompt. This allows us to also interpolate without conditioning on a prompt. We note that, expected without prompt, the intermediates points (the original images) but still realistic images are obtained as expected (except for using linear interpolation, discussion in Section and the interpolations yield smooth gradual changes.
not necessarily related to the end
23

---

Published as a conference paper at ICLR 2025
Figure 9: Without LOL transformation. The setup here is exactly the same in Figure 12 and Figure D respectively, except without the proposed (LOL) transformation (see Equation 3). The prompt used is "A high-quality photo of a parked, colored sports taken with DLSR camera with a 445.7MP sensor. The entire sports car is visible in the centre of the image. The background is simple and uncluttered keep the focus the sports car, with natural lighting enhancing its features." ". We note that the diffusion model does not produce images of a car without the transformation, and neither model produce anything else than visually the same image for all coordinates.
22
Figure 10: Additional slices of sports car subspace. The latents æı, Ts (corresponding to images) are converted into basis vectors and used to define a 5-dimensional subspace. The grids show generations from uniform grid points in the subspace coordinate system, where the left and right grids are for the dimensions {1, 2} and {3, 4}, respectively, centered around the coordinate for 21. Each coordinate in the subspace correspond to linear combination of the basis vectors. The flow matching model Stable Diffusion 3 (Esser et al. 2024) is used in this example. See Figure for another slice of the latent subspace.
H ADDITIONAL QUANTITATIVE RESULTS AND ANALYSIS
In Table 1 we show that our method outperforms or maintains the performance of the baselines in terms of FID distances and accuracy, as calculated using a pre-trained classifier following the evaluation methodology of Samuel et al. (2023). For an illustration of centroids and interpolations, see Figure 2 and Figure 16 respectively. The evaluation time of an interpolation path and centroid, shown with digit of precision, illustrate that the closed-form expressions are significantly faster than NAO. Surprisingly, NAO did not perform as well as spherical interpolation and several other baselines, despite using their implementation, which was outperforming these methods in Samuel
24

---

Published as a conference paper at ICLR 2025
interpelations
LERR
STLERO
NAO
LOL
Figure 11: Interpolation. Shown are generations from equidistant interpolations of latents Φ1 and 22 (in E [0, 1)) using the respective method. The diffusion model Stable Diffusion 2.1 (Rombach) et al. 2022 is used in this example.
Figure 12: Low-dimensional subspaces. The latents æi.... Us (corresponding to images) converted into basis vectors and used to define a 5-dimensional subspace. The grids show generations from uniform grid points in the subspace coordinate system, where the left and right grids are for the dimensions {1,3] and {2, 4}, respectively, centered around the coordinate for I1. Each coordinate linear combination of the basis vectors. The diffusion model Stable Diffusion 2.1 (Rombach et al. 2022) is used in this example.
in the subspace correspond
et al. 20233) We note one discrepancy is that we report FID distances using (2048) high-level features, while their work they are using (64) low-level features, which Seitzer 2020) is recommended against as it does not necessarily correlate with visual quality. In Table 3 we report results with all feature settings. We note that, in our setup, the baselines perform substantially better than reported including NAO in terms of FID distances using the 64 low-level feature
in Samuel et al. (2023) setting (1.30 in our setup vs 6.78 in their setup), and class accuracy during interpolation (62% vs 52%), which study below.
The number of DDIM inversion steps used for the ImageNet images is not reported in the NAO setup (Samuel et al. 2023), but if we use I step instead of 400 (400 is what use in all our experiments) the class preserving accuracy of NAO for centroid determination, shown in Table 4 resemble theirs more, yielding 58%. As we illustrate in Figure 5 and Figure [5] sufficient number of inversion steps is critical to acquire latents which not only reconstructs the original image but which be successfully interpolated. We hypothesise that much lower setting of DDIM inversions
25

---

Published as a conference paper at ICLR 2025
Figure 13: low-dimensional subspaces. The latents 21..... (corresponding to images) are converted into basis vectors and used to define 5-dimensional subspace. The grids show generations from uniform grid points in the subspace coordinate system, where the left and right grids are for the dimensions {1,3} and (2, 4), respectively, centered around the coordinate for æ1. Each coordinate in the subspace correspond to linear combination of the basis vectors. The diffusion model Stable Diffusion 2.1 (Rombach et al. 2022) is used this example.
steps used in their setup than in ours, which would be consistent with their norm-correcting optimization method having large effect, making otherwise blurry/superimposed images intelligible for the classification model.
Interpolation
Method
Accuracy
FID 64
FID 192
FID 768
FID 2048
Time
LERP
3.92%
63.4
278
2.18
199
6e 3s
SLERP
64.6%
2.28
4.95
0.173
42.6
9e 3s
NAO
62.1%
1.30
4.11
0.195
46.0
30s
LOL (ours)
67.4%
1.72
3.62
0.156
38.9
"s
Centroid determination
Method
Accuracy
FID 64
FID 192
FID 768
FID 2048
Time
Euclidean
0.286%
67.7
317
3.68
310
4e 4e-As
Standardized Euclidean
44.6%
5.92
21.0
0.423
88.8
le 3s
Mode norm Euclidean
44.6%
6.91
21.6
0.455
88.4
le 3s
NAO
44.0%
4.16
15.6
0.466
93.0
90s
LOL (ours)
46.3%
9.38
25.5
0.455
87.7
6e 6s
Table 3: Full table with class accuracy and FID distances using each setting in Seitzer 2020). Note
that FID 64, 192, and 768 is not recommended by [Seitzer] (2020). as it does not necessarily correlate with visual quality. FID 2048 is based on the final features of Inception (as in Heusel et al. (2017)) while FID 64, 192 and 768 are based on earlier layers; FID 64 being the first layer.
ADDITIONAL IMAGENET INTERPOLATIONS AND CENTROIDS
26

---

Published as a conference paper at ICLR 2025
1
interpllaions
82
SLBRP LERE
Figure 14: Interpolation during unconditional generation. Shown are generations from equidistant interpolations of latents and using the respective method. The latents æ and æ2 were obtained from DDIM inversion (Song et al. 2020a) with an empty prompt, and all generations in this example then carried out with an empty prompt. The larger images show interpolation index 8 for SLERP, NAO and LOI., respectively. The diffusion model Stable Diffusion 2.1 (Rombach et al. 2022) is used in this example.
Centroid determination of invalid latents
Method
Accuracy
FID 64
FID 192
FID 768
FID 2048
Time
Euclidean
42.3%
25.7
59.3
1.23
170
4e *s
Standardized Fuclidean
46.6%
3.35
16.6
1.23
175
le 1e
Mode norm Euclidean
50.6%
2.08
8.74
1.19
173
le 3s
NAO
57.7%
3.77
13.3
1.05
150
90s
LOL (ours)
48.6%
2.96
12.5
1.22
171
-4s
Table Centroid determination results we would use single step for both the DDIM inversion and generation (not recommended). This results in latents which do not follow the correct distribution Ni,, Σ) and which, despite reconstructing the original image exactly if generating also using single step, cannot be used successfully for interpolation, see Figure 5 As illustrated in the figure, interpolations of latents obtained through a single step of DDIM inversion merely results in the two images being superimposed. Note that the class accuracy is much higher for all methods such superimposed images, compared to using DDIM inversion settings which allow for realistic interpolations (see Ta) Moreover, the FID distances set to use fewer features lower for these superimposed images, while with recommended setting for FID distances (with 2048 features) they higher (i.e. worse), as would expect by visual inspection. The NAO method, initialized by the latent yielding the superimposed image, then optimises the latent yielding a norm close to norms of typical latents, which indicated in the table leads better "class preservation accuracy".. However, the images produced with this setting are clearly unusable as they do not look like real images.
27

---

Published as a conference paper at ICLR 2025
xtep
steps
sleps
steps
steps
steps
Image
LERE
SLERI
LOL
Imuge
Figure 15: Accurate DDIM inversions are critical for interpolation. Shown is the interpolation (center) of Image 1 and Image 2 using each respective method, after varying number of budgets of DDIM inversion steps (Song et al. 2020a). For each budget setting, the inversion was run from the beginning. We note that although a single step of DDIM inversion yields latents which perfectly reconstructs the original image such latents do not lead to realistic images, but merely visual "superpositions" of the images being interpolated.
28

---

Published as a conference paper at ICLR 2025
interpelations
LERP
SLERP
LOR NAD
Figure 16: Interpolation. Shown are generations from equidistant interpolations of latents æ and 22 using the respective method. The latents I1 and Ta were obtained from DDIM inversion (Song let al. (020a) of two random image examples from one of the randomly selected ImageNet classes ("Chesapeake Bay retriever") described in Section S The larger images show interpolation index for SLERP, NAO and L.OI., respectively. The diffusion model Stable Diffusion 2.1 (Rombach et al. 2022) is used in this example.
12
30g
Euclidean
Eaclidean
Ma
Euclidean
NAO
LOL
Figure 17: ImageNet centroid determination. The centroid of the latents æ1, U2, 33 as determined using the different methods, with the result shown in the respective (right-most) plot. The diffusion model Stable Diffusion 2.1 (Rombach et al. (022) is used in this example.
29

---

Published as a conference paper at ICLR 2025
2
3
317
2100
Buchddaa
Eaclidean
Ma. Eaclidean
NAO
LOL
Euclideon
Figure 18: Centroid determination of many latents. The centroid of the latents as determined using the different methods, with the result shown in the respective (right-most) plot. Ten and 25 latents are used the first two and second two examples (rows), respectively, and the plots show subset of these in the left-most plots. We noted during centroid determination of many latents (such as 10 and 25) that the centroids were often unsatisfactory using all methods; blurry, distorted etc. For applications needing find meaningful centroids of large number of latents, future work is needed. The diffusion model Stable Diffusion 2.1 (Rombach et al. 2022 is used in this example.
30


Reviews:
[{"review_id": "DpzMU9zOim", "reviewer": "Reviewer_Y6W4", "summary": "This paper proposed a novel method of linearly combining Gaussian latents of pretrained generative models (in the paper Diffusion model and flow matching model) that are simple, and shown in to be effective in linear interpolation and centroid determination tasks. The methodology is based on observation that latent samples from priors of pretrained generative models can lack Gaussian characteristics, which is the main reason for failure in the image reconstruction task.", "questions": "- Can COG perform rare-concept generation experiment similar to Samuel et al. (2023), section 5.1? I find the current empirical benchmarks that make comparisons with existing baselines quite lackluster.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Well-motivated problem, the proposed method is novel, reasonable, and most importantly, relatively simple, yet seems to performing quite well in (certain) empirical benchmarks.", "weaknesses": "- The writing of the paper can be improved. Some parts are ambiguously written. For example, the notion of latent space $x_T$ can be unified for both flow matching and diffusion at time $T$, instead of writing $x(0)$ and $x_T$ to avoid confusion. The important observation, starting at line 206: \"Having a latent vector with a norm that is likely for a sample from...\" is extremely unclear. In fact, the sentence does not make sense. \n- I have a big question mark about the usage of the univariate testing method in Section 3 (stated in 3.1 in between line 250-254). Why don't the authors use a multivariate hypothesis testing method that accounts for the correlation between features/pixels. This makes more sense statistically. \n- The authors mentioned that \"In this work we will propose a simple technique that guarantees that interpolations follow the latent distribution given that the original (seed) latents do, and which lets us go beyond interpolation\". However, I failed to find a Lemma/Theorem that stated such a theoretical guarantee of their method. \n- Empirical results seem questionable: the authors claimed to follow closely benchmark protocols from NAO paper of Samuel et al. (2023) (in fact the year of publication for this citation in the paper was stated wrongly -- please fix). More specifically, the results from Table 1 of the paper is very different from Table 1 in Samuel et al. (2023) -- the two tables are supposed to report results from the exact same experiment.\n\nDvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation. Advances in Neural Information Processing Systems, 37, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ePHNpE7OPc", "reviewer": "Reviewer_CkyB", "summary": "Generative models like diffusion/flow models can map data to Gaussian latent spaces, allowing additional manipulations. However, typical interpolation methods, like linear or spherical interpolation, often fail because they do not fully respect Gaussian distributional properties, leading to unrealistic generations. Current alternatives are either computationally costly or specific to certain models. This paper proposes Combination of Gaussian variables (COG), a simple, generalizable method that maintains Gaussian characteristics during interpolation and linear transformations, enabling effective latent manipulation across models and data types.", "questions": "1. I am confused by Figures 4 and 5. Is the 1-step inversion image correct? I expect an image of worse quality than a 2-step inversion following the trend. On this note, could you explain how you are doing the inversion, like one-step inversion and multi-step generation? Essentially, I am curious as to why a 1-step inversion reconstruction is this good, and the quality dips first and then becomes better as you increase the budget.\n2. The claim made in Section 3.2, \"stress some latents are rejected that lead to good reconstructions...\", needs to be backed up, in addition to the illustration in Figure 5. There should be more larger-scale quantitative evaluations. Even Figure 5 itself is not very convincing: why is the snake interpolation better than the clock one?\n2. Some Figures are not very readable. For example, the colored numbers in Figure 5, and the green crosses and red dots on the bottom row of Figures 4 and 5.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The core argument of the paper, \"the norm is not a sufficient statistic to look at when determining whether the latent is good for generations\", is intuitive and sound. Especially Figure 3 presents a convincing argument.\n2. They paper has many illustrations of the different interpolation methods results. In particular, the ones of low-dimensional subspaces are very nice.", "weaknesses": "1. The contribution of this paper is limited. Norm being not enough is known, and the proposed interpolation method, under the setting of diffusion/flow model, is just rescaling the samples norm: $\\alpha=0$ and we rescale the sample by the expected norm gain $\\sqrt{\\beta}$.\n2. The normality testing part feels out of place and under-explored. I assume it is there to justify that the norm is not enough, and we need broad characteristics. In that case, Figure 3 alone is enough. Then the paper never specifies what broad characteristics we should/could look at or optimize. Lastly but more importantly, the exploration done in Section 3 never went anywhere: as listed in point 1, the proposed method also does not look at the broad characteristics (essentially still just the norm since $y$ to $z$ is affine), and is not connected to the normality testing in any way.\n3. The actual advantages of the method are questionable. For example, there are no clear differences shown in Figure 1, compared to the (standardized/mode norm) Euclidean results. Also, I am wondering why COG is faster than Euclidean methods in Table 1?\n4. The writing of this paper can be improved. For example: (1) the writing on normality testing part is not clear. How exactly are you doing the normality test, as it requires a set of data points? (2) what setting is Table 1 for, SD or ImageNet Inversion, since there is only one FID per task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vSDBHbWVfG", "reviewer": "Reviewer_bMMA", "summary": "The paper presents Combination of Gaussian variables. It presents the theory behind the method and multiple experiments showing the effectiveness of the proposed method.", "questions": "- In Figure 5, it is hard to tell if an interpolation succeeded or failed. I can't really make any clear decision by looking at the images. It would maybe be better to show interpolation between things which are more intuitive such as human faces or animals.\n- Would it be possible to have a larger study of the correlation between the p-value and the Q-align values. For example, one can sample many examples and invert them using different budgets, then interpolate between pairs and check visual quality vs p-value, \n- In addition, some human evaluation would be good instead of the Q-Align metric.\n- I think that for table 1 it would be good to have a MOS evaluation of the different methods, FID is not a good enough metric.\n- In Figure 15, all interpolation methods seem to perform equally well. This makes me think that the proposed method is incrementally better at most. This is also the case in table 3.\n- What would happen if figure 6 for a different interpolation method? Would it fail?\n- There is no limitation section in the paper.\n- In figure 14 it seems the CoG generates an image which is remarkably close to image 2 rather than an interpolation of the two images. Could it be that the generated images are just one of the two options image1 and image2? It seems the interpolation is not really an interpolation at all.\n- Generally, I think most examples in the paper are not convincing and that examples that demonstrate the method better should have been chosen. For example, an interpolation between a dog and a cat is similar posture, or between two faces, or the same face with a different facial expression.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method is simple and easy to understand.", "weaknesses": "I think the examples in the paper are not convincing that the proposed method is much better than the baselines. Some numbers indicate this but there is no human evaluation in the results. I think that better examples would have made the paper stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VgoQ9XHJER", "reviewer": "Reviewer_ghwF", "summary": "This work examines the topic of finding low dimensional subspaces and semantically meaningful interpolant paths between the latent vectors of generative models, such as diffusion and flow models. The authors identify and validate normality tests that confirm their hypothesis that poor visual quality is a product of latent vectors being out of distribution for a model. They propose a simple and easy-to-implement approach for producing interpolants that lead to high quality generation.", "questions": "### Q1: In Figure 4, why is inversion with a higher budget worse than 1 step?\nColumns with budget 2-32 appear to be worse than a single step. Why is this the case?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written. The problem statement, background, identification of the issue, and proposed solution flow well and are easy to follow.\n\nThe visual evidence supporting the author’s proposed tests for identifying “failing” latents and for validating their proposed COG approach are compelling. Overall, the experimental validation is thorough and well executed.", "weaknesses": "The list below roughly follows the order of appearance in the manuscript. In my opinion W1, W2, W4, and W5 are the most important among these.\n\n### W1: The functional form / actual mechanics of the normality tests should be added as an appendix section\nFor readers less familiar with the actual tests employed, this would be very helpful. For example, I was only aware of the Kolmogorov-Smirnov test of univariate distributions and am not sure how the multivariate version is defined. I would definitely recommend adding these details as an appendix section to aid clarity of exposition.\n\n### W2: Figure 3 is hard to interpret\nI found this figure hard to digest. Specifically, are the images from columns 2 and 3 generated from the same latent vector but with different seeds? \n\n### W3: “Correlation” between acceptance of an inversion and reconstruction quality should be quantified.\nIn the second paragraph of Section 3.2, the authors state:\n> Figure 4 demonstrate a strong positive correlation between acceptance of an inversion (green)\n$\\mathbf{x}^{(*)}$ and its reconstruction quality,\n\nThis claim should be quantified if possible.\n\n### W4: Figure 5 is also difficult to interpret\nI found the bottom of Figure 5 also difficult to interpret. The relationship between Q-align score and acceptance under the normality tests does not appear to be clear cut and the confidence intervals seem quite wide.\n\n### W5: The evidence in Section 3.2 seems anecdotal\nSpecifically the claims made in the third paragraph of this section seem to only be supported by the few examples in the grid of Figure 5. If there were some way to better quantify / make the evidence for these claims more robust, that would make the claim about the efficacy of the normality tests much stronger in my opinion.\n\n### W6: Minor formatting issues\n- Line 153: The “Shoemake (1985)” citation should be `\\citep`\n- Line 158: The “Song (2020a;b)” citations should be `\\citep`\n- Line 245: The citation for Kolmogorov-Smirnov seems to be malformed: “An 1933” and should also be `\\citep`\n\n### W7: Formatting suggestions\nList below is not weaknesses per se, just some ideas I had about helping to make some of the figures easier to digest. Feel free to implement or ignore these: \n- In Figure 2, and other similar ones (e.g. Figure 6), perhaps place a red / some-colored border around the image in the first column that serves as the “anchor”/origin image and place a similar box around the corresponding image in the grid. Consider labeling the directions in the grid as well.\n- In the paragraph for section 3.1, at the end of the paragraph, I would recommend bolding the normality tests that the authors found to be most effective.\n- In Figure 5, it might be useful to also add the Q-align score as colored text to each image", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)