Conditional Generative Models are Provably Robust: Pointwise Guarantees for Bayesian Inverse Problems
Authors: Fabian Altekrüger
Paul Hagemann
Gabriele Steidl
Venue: Accepted by TMLR ()

Abstract: Conditional generative models became a very powerful tool to sample from Bayesian inverse problem posteriors. It is well-known in classical Bayesian literature that posterior measures are quite robust with respect to perturbations of both the prior measure and the negative log-likelihood, which includes perturbations of the observations. However, to the best of our knowledge, the robustness of conditional generative models with respect to perturbations of the observations has not been investigated yet. In this paper, we prove for the first time that appropriately learned conditional generative models provide robust results for single observations.

Published in Transactions Machine Learning Research (07/2023)
Conditional Generative Models are Provably Robust: Pointwise Guarantees for Bayesian Inverse Problems
Fabian Altekrüger
Department of Mothematics
Humboldi- Universilät zu Berlin
Paul Hagemann
hegemann@@mathh.tubbeer www.wthhww...bbbbii.c.mm
Institute of Mathematics Technische Uninersität Berlin
Gabriele Steidl
steidl@math..t berlin. de
Institude of Mathemalics Technische Universiläl Berlin
Reviewed on OpenReview: https: // openreview. net/ forum? id= Wcws061 for
Abstract
Conditional generative models became very powerful tool sample from Bayesian inverse problem posteriors. It is well-known in classical Bayesian lit- erature that posterior measures are quite robust with respect to perturbations of both the prior measure and the negative log-likelihood, which includes perturba- tions of the observations. However, to the best of our knowledge, the robustness of conditional generative models with respect to perturbations of the observations has not been investigated yet. this paper, we prove for the first time that ap- propriately learned conditional generative models provide robust results for single observations.
1 Introduction
Recently, the use of neural networks (NN) in the field of uncertainty quantification has emerged, as one often interested the statistics of solution and not just point estimates. particular, Bayesian approaches in inverse problems received great interest. In this paper, we are interested in learning the whole posterior distribution in Bayesian inverse problems by conditional generative NNs proposed, e.g., in (Adler & Öktem 2018 Ardizzone et al.) 2019 Batzolis et al. 2021 Hagemann et al., 2022) Addressing the posterior measure instead of end-to-end reconstructions has several advantages as illustrated in Figure More precisely, we consider a Gaussian mixture model as prior distribution and a lincar forward operator with additive Gaussian noisc, then the posterior density (red) can be computed explicitly. Obviously, these curves change smoothly with respect to the observation y, ie., we observe continuous behaviour of the posterior also with respect to obscrvations near zero. In particular, (samples of) the posterior can be used to provide

---

Published in Transactions Machine Learning Research (07/2023)
SMAP ZAnaE
ZMAP
Isaa
2
2
-0.5
0
-0.5
-1
-0.5
0.5
1
-1
-0.5
0.5
1 = 0.05
V = 0.01
1 = 0.01
y/ = 0.05
Figure 1: Posterior density (red), MAP estimator (blue) and MMSE estimator (green) for different observations y 00.55, 0001, 0.01, 0.05 (from left to right). While the MAP estimator is discon- tinuous with respect to the observation y, the posterior density is continuous with respect to The MMSE estimator just gives the expectation value of the posterior which is, in contrast to MAP, not the value with highest probability.
additional information on the reconstructed data, for example on their uncertainty. As can be seen in the figure, for fixed y the minimum mean squared error (MMSE) estimator delivers just an averaged value. In contrast to the non robust maximum a-posteriori (MAP) estimator, this is not the one with the highest probability. Even worse, the MMSE can output values which are not even in the prior distribution. For more comprehensive comparison of MAP estimator, MMSE estimator and posterior distribution, we refer to Appendix A
Several robustness guarantees on the posterior were proved the literature. One of the first results in the direction of stability with respect to the distance of observations was obtained by SSuart (2010) with respect to the Hellinger distance, see also Dashti & Stuart) (2017). A very related question instead of perturbed observations concerns the approximations of forward maps, which was investigated by Marzouk & Xiu 22009). Furthermore, different prior measures were considered in (Hosseini, 2017 Hosseini & Nigam 2017 Sullivan 2017), where they also discuss the general case in Banach spaces. Two recent works (Latz) 2020 Sprungk 2020) investigated the (Lipschitz) continuity of the posterior measures with respect to a multitude of metrics, where Latz (2020) focused on the well-posedness of the Bayesian inverse problem and [Sprungk] (2020) on the local Lipschitz continuity. Most recently, in Garbuno--nigo et al.) (2023) the stability estimates have been generalized to integral probability metrics circunventing some Lipschitz conditions done in Sprungk (2020). Our paper is based on the findings of |Sprungk (2220). but relates them with conditional generative NNs that aim to learn the posterior.
More prociscly, in many machine learning papers, the following idea is pursued in order to solve inverse problems simullaneously for all observations ye Consider a family of generative models Go(y:)) with parameters 0, which are supposed to map latent distribution, like the standard Gaussian one, to the absolutely continuous posteriors Y ie., Go(y, )) P2 Px In order to learn such conditional generalive model, usually a loss of the form
L(0) = Enyey DD(P Goly ) Pz)]
chosen with some "distance" between measures like the Kullback-Leiin (KL.) divergence D KL used in Ardizzone al. (0019) for training normalizing flows the Wasserstein-1 distance
D = W, appearing, c.g., in the framework of (conditional) Wasscrstein generative adversarial
2

---

Published in Transactions Machine Learning Research (07/2023)
networks (GANs) [Adler & Öktem 2018 Arjovsky al. 2017 Liu et al. 2021 Also conditional
diffusion models Igashov et al. 2022: Song al. 2021b Tashiro et al. 2021 fit into this framework.
Here De Bortoli 2022) showed that the standard score matching diffusion loss also optimizes the Wasserstein distance between the target and predicted distribution.
However, in practice we are usually interested in the reconstruction quality from a single or just a few measurements which are mull sets with respect to Py. In this paper, we are interested in the important question, whether there exist any guarantoes for the NN output to be close the posterior for one specific measurement y. Our main result in Theorem | shows that for NN learned such that the loss becomes small in the Wasserstein-1 distance, say L(Ø) < E, the distance Wi Wi WP(xxx== Go +) Pz) becomes also small for the single observation y. More precisely, get the bound
Pz) Wi (P Go Pz) S Centr,
where C is constant and n is the dimension of the observations. To the best of our knowledge, this is the first estimate given in this direction.
We like to mention that in contrast to our paper, where we assume that samples are taken from the distribution for which the NN was learned, Hong et al. (2022) observed that conditional normalizing flows are unstable when feeding them out-of-distribution observations. This is not too surprising given some literature on the instability of (conditional) normalizing flows (Behrmann al., 2021 Kirichenko et. al., 2020
Outline of the paper. The main theorem is shown in Section For this we introduce several lemmata for the local Lipschitz continuity of posterior measures and conditional generative models with respect to the Wasserstein distance. In Section 3 we discuss the dependence of our derived bound on the training loss for different conditional generative models. In Appendix A we illustrate by a simple example with a Gaussian mixture prior and Gaussian noise, why posterior disttributions can be expected be more stable than maximum a-posteriori (MAP) estimations and have more desirable propertics than minimum mcan squared error (MMSE) cstimations.
2 Pointwise Robustness of Conditional Generative NNs
Let X E R. be a continuous random variable with law Px determined by its density function px and f: Rmm -> R= measurable function. We consider a Bayesian inverse problem
noisy(f(X))
(1)
where 'noisy' describes the underlying noise model. A typical choice is additive Gaussian noise, resulting in
Y= f()++E,, o0,,
Let =; + R be conditional generative model trained to approximate the posterior distribution PxIY= using the latent random variable Z E Rd, We will assume that all appearing measures are absolutely contimuons and that the first moment of G(y. )) P2 is finite for all y E R". In particular, the posterior density is related via Bayes' theorem through the prior px and the likelihood pr]x= as
Yx X PY|X==Px:
3

---

Published in Transactions Machine Learning Research (07 (07/2023)
where o means quality up to a multiplicative normalization constant. Since the posterior density Px|Y=y is only almost everywhere unique, we choose the unique continuous representative. Further, we assume that the negative log- log-iikelhhodd - log Py|x=a is bounded from below with respect to 2, i.e, inf,- log X > 00. In particular, this includes mixtures of additive and multiplicative noise Y = f(x) + + Ex)()), X,, Ei and E are independent, or log-Poisson noise commonly arising in computerized tomography.
We will use the Wasserstein-1 distance (Villani 2009), which is a metric on the space of probability measures with finite first moment and is defined for measures jr and on the space R as
Wi (1, inf [ xEll(a)) RmM =
| (1, v),
xB
where Ij v) contains all measures on R x Rd with and v as its marginals. The Wasserstein distance can be also rewritten by its dual formulation (Villani 2009 Remark 6.5) as
/ ((z)d(m-v)))) - (2))
(1((v)) = max Lip(p)s)
(2)
First, we show the local Lipschitz continuity of our generating measures Gly, ) Pz with respect to where we assume a local boundedness of the Jacobian of the generator with respect to the observation.
Lemma 1 (Local Lipschitz continuity of generator). For all r > O, there exists some Lr > such that for any parameterized family of generative models G with [V, I L, for all
2 E supp(Pz) and all y e IR" with ||yl| $ r it holds
Wi ) Pz, G(y2, :z) S Lr y/ll
for all y1, U2 R with 11ya ll. I1 s r.
Proof. We the mean value theorem which yields for every a supp(Pz) and all yn y2 E R with || Il, ||ve|| Sr
[IG(y, G(yz, =)]l = 11 L v,G(m + (uz2 v1), 23n - s2)de
< / IV + v1), =)|(df|lv1 22ll
$ Lr 1/ll.
Next, we apply the dual formulation of the Wasserstein-1 distance to estimate
Wi WG(G )
- Pz. G(z2.))+P2)
= max E Ben 2)) ((G( =))] Lip(e)s1
- max E (lp(p( 2)) (GG((2 2)) Lip(e)s1
< Ez-Py Pz [I|G((), G(yz, =)[l
< 1l1 - 12ll-
4

---

Published in Transactions Machine Learning Research (07/2023)
We want highlight that there is trade-off between regularity of the generator due to small Lipschitz constant (Gouk et al. 2021 Miyato et al., 2018) and expressivity of the generator requiring a large Lipschitz constant (Hagemann & Neumayer 2021 Salmona et al. 2022
Remark 2. If Pz is supported generators which are, e.g., continuously differentiable and then it follows from the extreme value theorem. Note that if we choose Pz to be a Gaussian distribution, then it holds supp(Pz) = R Thus, for continuously differentiable generators is not clear that this assumption is fulfilled, but at least the weaker ussumpliio [VV,Gyy, =]]l S Tr for all Rd with ||=|| S and all y E IR with Ily|l S holds true. In this case, we can show that Lemma 7 holds true up to on arbitrary small additive constant, see Appendir B for more details.
compact sel, then the assumplion in Lemma is fulfilled for
Corollary 19) for Euclidean spaces, the local Lipschitz continuity of the posterior distribution with respect to the Wasserstein-1 distance is guaranteed under the assumption of a locally Lipschitz likelihood.
By the following lemma, which is just (prungk 2020
Lemma 3 (Local Lipschitz contimuity of the posterior). Let the forward operator f and the likelihood Py X== in [ be measurable. Assume that there erists a function M: (0) x R + 00) which monotone in the first component and non- dec-asccgg in the second component such that for all R= with 11 |122l S r for and for all =R holds
y1 1/2
PY (12)
S (r, D1 -22l-
(3)
Furthermore, assume that M (r, -I- E L2p (RT .R). Then, for any myyr= O there exists a constant C, < 0o such that for all yı, y2 E I=" with | |l, |1 S r we have
W P
S Crllv1 - y2ll.
The Lipschitz constants of the family of generative models and the posterior distributions Px|y= can be related to each other under some convergence assumptions. Let the assumptions of Lemma be fulfilled, assume further that there exists a family of generative models (G*), satisfying
lim G (y, )4 Pz PxIY= c-
with respect to the Wi-distance and consider observations y1, y2 R" with ||l l1sall S Then, by the triangle inequality it holds
lim W1(G (31, 4) Pz, G (y2, ) P2) s lim W1(G )4 c+0
P2, P2, Px(V= (x
=-0
+ +W, + G G= :) Pz)
(P Px)x=
I C.lls1 y2ll-
Hence, under the assumption of convergence, generative models to behave similar to the one of the posterior distribution. Remark 4. The assumption (3) is for instance fulfilled for additive Gaussian noise 3 ~ (N In this case
expect the Lipschitz constant of our conditional
-oogPy
n log (og(2TT?)
202 lV f(x)11
5

---

Published in Transactions Machine Learning Research (07/2023)
Hence log PY(x=)())) is differentiable with respect to y and we get local Lipschitz continuity of the negative log-likelhhood.
Now can prove our main theorem which ensures pointwise bounds on the Wasserstein distance between posterior and generated measure, if the expectation over Py is small. In particular, the bound depends on the local Lipschitz constant of the conditional generator with respect to the observation which may depend on the architecture of the generative model, the local Lipschitz constant. of the inverse problem, the expectation over Py and the probability of the considered observation y. We want to highlight that the bound depends on the evidence py py( an observation y and indicates that we generally cannot expect good pointwise estimate for out-of-distribution observations, i.e,, pr (y) 0. This in agreement with the empirical results presented in Hong et al. 20022 The proof of Theorem 5 has nice geometric interpretation, which is visualized in Figure
Theorem 5. Let the forward operator and the likelihood pv in 1 fulfill the assumptions of Lemma Let R" be an observalion. wilh pr >> 0. Further, assume that y + pr(y) is differentiable with [V S. K for K > O and all 12", For fixed k = 2K +llill, assume that we have trained family of generative models G which fulfills VV, 2)ll - Lx for all E supp(Pz) and all y E IR" with |yyl| S k for some Ly > such that
By (Wi(PxIX= G(y, )) Pz)) se
(4)
for some & > Then we have for e S that
28 n+1
G(j. -) Pz) S (Ix + Ck)
(5)
2K
Sn 2k )r
where Sn x /T 1) and C. the Lipschitz constant from Lemma
If additionally satisfies
x+1 +c. then it holds 2n
1
()(20
2n
m=1
Wi(P
S (Lx + Cc)² nt. (1 n
8
(6)
S,a
Proof. Let 0 <rr a such that
Then, for E (T),), there exists by the mean value theorem some ç E yy
2K
1pr - py (0) < Vpy (E)llly - -l = Kr S 2
Consequently, each y E (D) has at least probability py 2 2. Moreover, by the volume of the n-dimensional ball holds that
Pv(B(()) = (.(
2 py(yddy
π
a 2
S r"
a 2
r 7
Now claim that there exists ① B,) with
2e
W1 x G(T, P2)
(7)
S nd
6

---

Published in Transactions Machine Learning Research (07 (073)0
If this would not be the case, this would imply a contradiction to by
By (P G(y, ) Pz)]
Wi (P G(u, ) Pz)dPr(1)
2
( ) P2)
B,(
23
dPy(y)
Suraa
B,(
2
= (B,
2 6.
Spr
Next, we show the local Lipschitz continuity of + Wi Wi (P Gly, ( Pz) on B, (ū) by combining
Lemma [ and Lemma 3 Let y1, U2 E (ū), that 11 1l. 1 |lóll r. Let L 0 be the
local Lipschitz constant from Lemma [ and Clsltr the local Lipschitz constant from Lemma Using the triangle inequality and its reverse, we get
IW1 [Px G(v1.)) P2) - W1(Px x G(v2, + P2)l
I [Wi(Px ((11::)+P) - WP(Pxx G(Uz, P2))
+ (PxIx== G(yz, )2 > Pz) - Wi Wi(PxxX G(yy, ) Pz)
(8)
< Wi(G( Pz, G(y2.) + (Px Px
$ (L M++ + ( -22ll-
Combination of the results in and g
yields the estimate
Wi(Px y. G(ū. ) Pz) ( G(ū.) - Wi(P
x (P G(T.)) + P2))
WW xI [ G(ū, Pz)l
28
2 (L fil + )1ill+rrr+++ Snraa
2e
S (L + ClsSl++ ate )r+++
Sa
i S. 1, we can choose r = gn for which the right-hand side bocomes minimal, given by
2K S 2K
which results in
5
On the other hand, the radius r,
2ns
WFI
(L
G 2
Sna
Plugging this in, we get 6 which has the same asymptotic rate.. However, we need that r S 2k which implies
a
(Lnjsl+ d
Clall zi )Saa
E E
2K
2n
Note that in 4 we assumo that the expectation over Py of the Wasserstein distance is small. When training a generator, usually finite training set is available. The measure Py can be approximated by the empirical training set with rate n 1ld on compact sets, where is the size of the training set and d the dimension, see, c.g., Weed & Bach (2019.
7

---

Published in Transactions Machine Learning Research (07/2023)
supp(Py)
Wi
(7)
W1 - (L - Crr
2c 3gr"a
Figure 2: Geometric interpretation of the proof of Theorem Inside the ball Br) find some for which the Wasserstein distance W1 Wi G(ğ. P2) is bounded by 3.r"a larily of the generator and of the inverse problem, the Wasserstein distance Wi (Px Y=g. G(ū. 1)# Pz) can be bounded by the triangle inequality.
28 Using the regu-
Remark 6. We can get rid of the dimension scaling g by choosing the radius as r yields
a which
2K
2
(P (4T S (1 + Clsll++ zi 2K
+ Sn
2K )"a
This comes at the disadvantage that the first term is constant with respect to
The following corollary provides characterization of perfect generative model. If the expectation goes to zero, then for all y R with (v) > 0 the posteriors Px|y= get predicted correctly. Corollary 7. Let the assumptions of Lemma 1 and Lemma hold true and assume global Lipschitz constant in Lemma 7 Let py be differentiable with || V/ (y)l < K for some K > and all IR". Consider a fumily of generalive networks (Ge) )e>0 fulfilling
Ey-Py [Wi (Px(Px (2))
and assume that the Lipschitz constants L of G from Lemma ! are bounded by some L < co. Then for all observations y E Rn" with py > it holds
Wi WP G (y, )) Pz)
+
Proof. We can assume that 1, then the statement follows immediately from Theorem 5
Finally, we can use Theorem 5 for error bounds of adversarival attacks on Bayesian inverse problems.
Following the concurrent work of Gloockler et al. (2023). adversarial attack conditional

---

Published in Transactions Machine Learning Research (07/2023)
generative model consists in finding a perturbation to the observation ÿ so that the prediction of the conditional generative model is as far away as possible from the true posterior, i.c.,
8 = arg max Wi (P G(ğ 0,
Note that Gloeckler et al. 2023 use the KL divergence for their adversarial attack, but for our theoretical analysis the Wasserstein distance is more suitable.
The following corollary yields a worst case estimate on the possible attack. In the case of imperfect trained conditional generative models the attack can be very powerful depending on the strength of observation and the Lipschitz constant of the generator. Tf the conditional generative model is trained such that the expectation in (4 is small, then the attack can only be as powerful the Lipschitz constant of the inverse problem allows.
Corollary 8. Let the forward operator f and the likelihood pr|x=r in L fulfill the assumptions
of Lemma 3 Let ÿ E R and Rn with [|ø]] I be an observation with py + 5) = a 0. Further, ussume that y (y) is differentiable with [[Vry(y)] < K for K > 0 and all y RR.. For fized l 2 + lliil + B, assume that we have trained family of generative models G which fulfills (Vy,)( La for all E supp(Pz) and all R= with |lyl| I k for some Ly such that
Py [W1 (Xx (( P2))
for some 0 <E S 1. Then we have the following control on the strength of the adversarial attack
gntia
2g
W Pz) Clsl +B (Lx Ck)
2K
a Sn
2K
Proof. Since it holds
( Px( + - G(ū 6,
the result follows from the application Lemma g and Theorem
3 Conditional Generative Models
In this section, we discuss whether the main assumption, namely that the averaged Wasserstein distance
P2)) in 4 becomes small, is reasonable for different condi- tional generative models. Therefore we need to relate the typical choices of training loss with the Wasserstein distance. For a short experimental verification in the case of conditional normalizing flows we refer to Appendix C
In the following we assume that the training loss of the corresponding models become small. This can be justified by universal approximation theorems like (Teshima al. 2020 Lyu et al., 2022 for normalizing flows or [Lu & Lu 2020 for GANs.

---

Published in Transactions Machine Learning Research (07/2023)
3.1
Conditional Normalizing Flows
Conditional normalizing flows [Alterrüger & Hertrich 2023 Andrle et al. 2021 Ardizzone al., Winkler et al. 2019) are family of normalizing flows parameterized by condition, which in our case is the observation y- The aim is to learn network : IK" x - Rmm such that T(v. -) is diffeomorphism and Tv, )) Pz z Pxiy=y for all y E RK, where s means that two distributions are similar in some proper distance or divergence. This can be done via minimizing the expectation on of the forward KL divergence Ey-Py [KL(Px|==y: ( P2)), which equal, up to a constant, to
2019
Ex-Px I- loggoo(TTT (y, x)) log( det DT (y, (u, x)D],
where the inverse is meant with respect to the second component, see Hagemann et al. (2022) for more details. Training a network using the forward KL has many desirable properties like a mode-covering behaviour of T(, .) Pz. Now conditional normalizing flows are trained using the KL divergence, while the theoretical bound in Section 2 relies on the metric properties of the Wasserstein-1 distance. Thus we need to show that we can ensure small e in (4) when training the conditional normalizing flow as proposed. Following Gibbs Su (2002) Theorem 4), we can bound the Wasserstein distance by the total variation distance, which turn is bounded by KL via Pinsker's inequality (Pinsker 1963). i.e,,
Py (PP())) T(v,)) p By Byypy [TV( -T(,, )) Pz)
s \22 By [KL [KL( T(y, -) Pz)l:
where C is constant depending on the support of the probability measures. However, by definition supp(T(0, ) R By Altekrügge et al. (2023 Lemma 4) the density (T Pz decays exponentially. Therefore, we expect in practice that the Wasserstein distance becomes small if the KL vanishes even though (iibss Su 2002 Theorem 4) is not applicable.
3.2 Conditional Wasserstein GANs
In Wasserstein GANs (Arjovsky et al. 2017). a generative adversarial network approach is taken order to sample from a target distribution. For this, the dual formulation (2) is used in order to calculate the Wasserstein distance between measures Px and Py. Then the 1-Lipschitz function reinterpreted as discriminator in the GAN framework (Goodfellow al.) 2014). If the correspond- ing minimizer in the space of 1-Lipschitz functions can be found, then optimizing the adversarial
Wasserstein GAN loss directly optimizes the Wasserstein distance. The classical Wasserstein GAN loss for a target mcasure j and a generator G: Rd + Rm is given by
min max Ex-Px px [x(x) f(G(z)), 9 Lip(p)S1
where E N is the dimension of the latent space.
The Wasserstein GAN framework can be extended to conditional Wasserstein GANs (Adler & Liu et al. 2021) for solving inverse problems. For this, we aim to train generators R. and average with respect to the obsorvations
Öktem 2018
G: R
L(0) = Ey- max E (x)()) (G (G((,))))]]
Táp(fy)S1
Hence minimizing this loss (or variant of it) directly enforces small in assumption
10

---

Published in Transactions Machine Learning Research (07/2023)
3.3 Conditional Diffusion Models
In diffusion models, a forward SDE, which maps a data distribution to an approximate Gaussian distribution considered (Song al. 2021a b). Then the theory of reverse SDEs (Anderson 1982 allows sample from the data distribution by learning the score V Prz), where (c(x) is the path density of the forward SDE. The forward SDE usually reads
dx, axxd V2adW.,
while the reverse SDE is given by
dY, -aydt V llogp((z)dt + VadW..
where R describes the schedule of the SDE. However, the path density Pa(x) is usually in- tractable, so that the score V Por) is learned with NN so: [0, 7] I + In such that st x) x v (ogg:(()) for all t E [0, and x E RT. This can be ensured using the so-called score matching loss (Song et al. 2021b) defined by
min E)- [[]so(t, x) po
In order to solve inverse problems, we can consider a conditional reverse SDE
dY, = aYYdt Vlogp( + 22ddW,
where Pr (xly) the conditional path density given an observation E R" Consequently, we con- sider conditional diffusion models, where a NN 88: IK x [0, x - Rmm learned to approximate so sy,,t, v (ogg:(x)))) for all E [0, T], Rm and all observations matching loss for conditional diffusion models is given in Batzolis et al. 2021 Theorem 1) as
R= Then the score
I(()= Ey--y
(lso V (og))())y)11)
Denote by Y the solution to the approximated SDE starting at Yo ~ Pz and ÿv the solution of the approximated SDE conditioned on an observation E R", Then we can use the bound derived in Pidstrigach et al. 2023 Theorem 2) which gives
EyyPy [2x Pey)l Bympy [C W2 (Px + TL((),
where C' is constant depending the length of the interval T and the Lipschitz constant of the conditional score V Vlogp.<zl)) Finally, Hölders inequality yields for the Wasserstein-1 distance
Ey [1 (P Ppy)l s By~ry [C (Pxg +T((),
Hence, when training the conditional diffusion model by minimizing we also ensure that. (4) becomes small. For more in depth discussion with less restrictive assumptions on the score, see also (De Bortoli 2022).
11

---

Published in Transactions Machine Learning Research (07/200 (07/2023)
3.4 Conditional Variational Autoencoder
Variational Autoencoder (VAE) (Kingma & Welling 2013 aim approximate distribution Px by learning stochastic encoder Es: R R x Rd determining parameters of the normal distribution (Mp(1)), (x)) for sampled from Px and pushing Px to a latent distribution Pz with density pz of dimension d E N. In the reverse direction, a stochastic decoder Da: R - 1 x m determines parameters of the normal distribution to Px. By definition, the densities of Es and Da are given by (0(2(z) = (s))) Eo()) and po(2)2) (x; po(2), (z)), respectively. These networks are trained by minimizing the so-called evidence lower bound (ELBO)
(z)) for & E R and pushes Pz back
ELBO(0, 6) Px (19) ( (z1z)pz(z)) (=]()(
By Hagemann et al.
20233 Theorem 4.1), the loss L(0, Φ) related to KL by KL(Px, Da# P2) < ELBO Φ).
We can solve inverse problems by extending VAËs to conditional VA (Lim et al. 2018 Sohn ct al. 2015) and aim to approximate the posterior distribution Pxiy=y for given observation y E R". The conditional stochastic encoder E I² x R IRd x Rdd and conditional stochastic decoder Do: IR" RRd + Rm x Rmm are trained by
I(0, ()) = By P [Eaggs(-laa)) [og(ps 2 =) - (=l(,
By the same argument as above, the KL can be bounded by
Ey-ry [KI(PX Do(y, ) P2)1 - ((0,0)
and, using similar arguments as in Section 3.1 we get. the estimate
C Ey WWx Doly, P2) v2 L(0,0).
4 Conclusion
We showed a pointwise stability guarantee of the Wasserstein distance between the posterior Px|Y= of Bayesian inverse problem and the learned distribution G(y,,)aa Pz of conditional generative model G under certain assumptions. In particular, the pointwise bound depends on the Lipschitz constant the conditional generator with respect to the observation, the Lipschitz constant of the inverse problem, the training loss with respect to the Wasserstein distance and the probability of the considered observation.
The required training accuracy of the bound depends the Wasserstein-1 distance between the target distribution and the learned distribution. However, some conditional networks the con- ditional normalizing flow are not trained to minimize the Wasserstein-1 distance. Consequently, direct dependence of the bound on the training accuracy with respect to the KL divergence would be helpful. Under very strong assumptions, the continuity in Lemma [ has been shown for KL in Baptista et al. 2023). This could be used to derive a similar statement.
Furthermore, our bound is a worst case bound and is not always practical the constants are large. It would be interesting to check whether tightness of the bound can be shown for some examples.
12

---

Published in Transactions Machine Learning Research (07/2023)
Acknowledgement
P.H. acknowledges funding by the German Research Foundation (DFG) within the project of the DFG-SPP 2298 "Thcorctical Foundations of Deep Learning" and F.A. within project EF3-7 of Germany's Excellence Strategy The Berlin Mathematics Research Center MATH+ The authors want to thank B. Sprungk for posing the question considered this paper, namely there are guarantees that conditional generative NNs work well for single observations. Moreover, many thanks to J. Hertrich for fruitful discussions and for suggesting the example in the appendix.
References
Jonas Adler and Ozan Öktem. Deep Bayesian inversion. arXiv preprint arXio:1811.0501 2018
Fabian Altekrüger and Johannes Hertrich. WPPNets and WPPFlows: The power of Wasserstein patch priors for superresolution. STAM Journal on Imaging Sciences, 16(3):1033-100 1067, 2023.
Fabian Altekrüger, Alexander Denker, Paul Hagemann, Johannes Hertrich, Peter Maass, and Gabriele Steidl. PatchNR: learning from very few images by patch normalizing flow regular- ization. Inverse Problems, 39(6), 2023.
Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 13133331,,326, 1982.
Anna Andrle, Nando Farchunin, Paul Hagemann, Sebastian Heidenreich, Victor Soltwisch, and Gabriele Steidl. Invertible neural networks versus MCMC for posterior reconstruction in grazing incidence x-ray fluorescence. In Abderrahim Elmoataz, Jalal Fadili, Yvain Quéau, Julien Rabin, and Loic Simon (eds.), Scale Space and Variational Methods in Computer Vision, pp. 528 539. Springer International Publishing, 2021.
Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Guided image generation with conditional invertible neural networks. arXiv preprint arXiv:1907.02399, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference Machine Learning, pp. 214-223. PMLR, 2017.
Ricardo Baptista, Bamdad Hosseini, Nikola B. Kovachki, Youssef M. Marzouk, and Amir Sagiv. An approximation theory framework for measure-transport sampling algorithms. arXiv preprint arXim:2302. 13965, 2023.
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Elmann. Conditional image generation with score- based diffusion models. arXiv preprint arXiv:2111.1360 2021.
Jens Behrmann, Paul Vicol, Chin- Wang, Roger Grosse, and Joern -een-- Jacobsen. Under- standing and mitigating exploding inverses in invertible neural networks. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intel- ligence and Slatistics, volume 130 of Proceedings of Machine Learning Research, pp. 1792- 1800. PMLR, 2021.
M. Dashti and A. M. Stuart. The Bayesian approach to inverse problems. In Handbook of Uncer- tointy Quantification, pp. 311-428 Springer, 2017.
13

---

Published in Transactions Machine Learning Research (07/2023
Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis.
Transactions Machine Learning Research, 2022.
Alfredo Garbuno-Inigo, Tapio Helin, Franca Hoffmann, and Bamdad Hosscini. Bayesian posterior perturbation analysis with integral probability metrics. ar preprint ar wxix:2303.01512 2023.
Alison L. Gibbs and Francis Edward Su. On choosing and bounding probability metries. Interna-
tional Statistical Review / Revue Internationale de Statistique, 70(3):419-433 2002.
Manuel Gloeckler, Michael Deistler, and Jakob H. Macke. Adversarial robustness of amortized Bayesian inference. In Andreas Krause, Emna Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), Proccedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11493 11524. PMLR, 23-29 29 Jul 2023.
lan Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Y. Bengio. Generative adversarial networks. Advances in Neural Informa- tion Processing Systems, 3, 2014.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J. Cree. Regularisation of neural networks by enforcing Lipschitz continuity. Machine Learning, 110(2))393-416 2021.
Dario Grana, Torstein Fjeldstad, and Henning Omre. Bayesian Gaussian mixture linear inversion for geophysical inverse problems. Mathematical Geosciences, 49(4):493-51x) 2017.
P. Hagemann, J. Hertrich, and G. Steidl. Generalized Normalizing Flows via Markov Chains. Elements in Non-local Data Interactions: Foundations and Applications. Cambridge University Press, 2023.
Paul Hagemann and Sebastian Neumayer. Stabilizing invertible neural networks using mixture models. Inverse Problems, 37(8). 2021.
Paul Hagemann, Johannes Hertrich, and Gabricle Steidl. Stochastic normalizing flows for inverse problems: A Markov chains viewpoint. STAM/ASA Journal on Uncerlainly Quanlificalion, 10 (3):1162-1199) 1190, 2022.
Seongmin Hong, Inbum Park, and Se Young Chun. On the robustness of normalizing flows for inverse problems in imaging. arXiv preprint XrXix:2212 04319, 2022.
William C. Horrace. Some results on the multivariate truncated normal distribution. Journal of Multivariate Analysis, 94(1):209 221, 2005.
B. Hosseini. Well-posed bayesian inverse problems with infinitely divisible and heavy-tailed prior measures. SIAM/ASA Journal on Uncertainty Quantification, 5(1):1024-1060, 2017.
B. Hosscini and N. Nigam. Well-posed Bayesian inverse problems: priors with exponential tails. STAM/ASA Journal on Uncerlainly Quanlification, 5(1):436-444 465, 2017.
Ilia Igashov, Hannes Stärk, Clément Vignac, Vietor Garcia Satorras, Pascal Frossard, Max Welling, Michael Bronstein, and Bruno Correia. Equivariant 3d-conditional diffusion models for molecular linker design. Xiv preprint ar arXim:2210.066 10.007274, 2022.
14

---

Published in Transactions Machine Learning Research (07/2023
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational 1rX1::1812.6114, 2013.
bayes.
arv preprint
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of-distribution data. In Advances in Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc.
Jonas Latz. On the well-posedness of Bayesian inverse problems. SIAM/ASA Journal on Uncer- luinly Quantification, 8(1):451 14241-482,,. 2020.
Jaechang Lim, Seongok Ryu, Jin Woo Kim, and Woo Youn Kim. Molecular generative model based on conditional variational autoencoder for de novo molecular design. Journal of cheminformatics, 10(1):1-9, 9, 2018.
Shiao Liu, Xingyu Zhou, Yuling Jiao, and Jian Huang. Wasserstein generative learning of condi- tional distribution. Xiviv preprint or arXiv:2112 12..122227 10039, 2021.
Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advonces in Newral Information Processing Systems, volume 33, pp. 3094 3105. Curran Associates, Inc., 2020.
Junlong Lyu, Zhitang Chen, Chang Feng, Wenjing Cun, Shengyu Zhu, Yanhui Geng, Zhijie Xu, and Yongwei Chen. surrogates. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
-h-universal diffcomorphism approximators as superior neural
Y. Marzouk and D. Xiu. A stochastic collocation approach to Bayesian inference in inverse problems. Communications in Computational Physics, 6(4):826 847, 2009.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-dimensionall diffu- sion models for function spaces. arXiv preprint arXiv:2302. 10130, 2023.
MS Pinsker. Information and information stability of random quantities and processes. Technical report, Foreign Technology DIV Wright-Patterson AFB, Ohio, 1963.
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnès Desolneux. Can push-forward gen- erative models fit multimodal distributions? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (ods.), Advances in Neural Information Processing Systems, 2022.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnell (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Newral Information Processing Systems, Montreal, Quebec, Canada, pp. 3483 3491, 2015.
15

---

Published in Transactions Machine Learning Research (07/2023)
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021a.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Represenlations, 2021b.
Björn Sprungk. On the local Lipschitz stability of Bayesian inverse problems. Inverse Problems, 36 (5), 2020.
A. M. Stuart. Inverse problems: A Bayesian perspective. Acta Numerica, 19:451-559, 2010.
T. J. Sullivan. Well-posed Bayesian inverse problems and heavy-tailed stable priors. Inverse Problems in Imaging, 11(5))857-877 874, 2017.
space
G. M. Tallis. Elliptical and radial truncation in normal populations. Annals of Mathematical Statistics, 34:940-944, 1963.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermonn CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-basee invertible neural networks are universal diffeomorphism approximators. In Pro- ceedings of the 34th International Conference on Newral Information Processing Systems, Van- couver, BC, Canada, 2020.
Cédric Villani. Optimal Transport: Old And New, volume 338. Springer, 2009.
Jonathan Weed and Francis Bach. Sharp asymptotic and finite-sample rates of convergence of empirical measures Wasserstein distance. Bernoulli, (5(4A) ):260 2648, 2019.
Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likelihoods with conditional normalizing flows. Xiv preprint Xic: 00042, 2019.
A Example the Robustness of the MAP and Posterior
We like to provide an example that illustrates the stability of the posterior distribution in contrast the MAP estimator and highlights the role of the MMSE estimator.
By the following lemma, see, e.g., Grana et. al. 2017 Hagemann et al. 2023 the posterior of a Gaussian mixture model given observations from a linear forward operator corrupted by white Gaussian noise can be computed analytically.
Lemma 9. Let X Ek wpN(m,, Σx) E R be Gaussian minture random variable. Suppose
that
== AX+E.
16

---

Published in Transactions Machine Learning Research
where A 12 18 is lineor operator and E N(0, In). Then the posterior also a Gaussian
minture
Px|Y=y
Wy W Σx)
with
= = 12=42AA+++
mi Ex y Ex Mx)
and
Tx = wk exp
2
(ma m. mz mx2iz'ma)
Now, for some small we consider the random variable E R with simple prior distribution
1 N(-1, + Px
2
and observations from Y = X + with noise ~~ N((, The MAP estimator is given by
SMAP(U) E arg max px|Y=
arg min
20 (3) - 12)2-log
log
1
(e 202(0-1)² =
ata (ott)))))
2
1
arg min
20 (y x²²
(x² +1) 2ºa
log (oosh 25)
The above minimization problem has unique global minimizer for y 0 which computed numerically. Figure 3 (top) shows the plot of the function ΣMAP(у) for e² 0.052 and different values of a. Clearly, small perturbations of near zero lead to qualitatively completely different r-values, where a smaller noise level a lowers the distance between the values TMAP(у) for > 0 and < O. In other words, the MAP estimator not robust with respect to perturbations of the observations near zero.
In contrast, using Lemma we can compute the posterior
1 w1 + 00
Pxiy=
(wi (wIN(111, 0²) 22 (-1120)))))
with
02g2
2²y d²
2²y d²
x²_
mi
m2 =
x² + 22
x2 02
x2 + o2
1
(s"y + 02)2 -
1
(e2 + 02) 1))
(E 0²)²
W1 2
exp
2g
(e2 ( + 02)
W2 28
exp
2e
17

---

Published in Transactions Machine Learning Research (07/2023)
0.5
0.5
0.5
SMAP
JMAP
-0.5
-0.5
-0.5
-0.5
0.5
-0.5
1
1
35085
0
-1
-1
-1
-0.5
0
0.5
-1
-0.5
0.5
-1
-0.5
0
0.5
8
y
0²
0.01
0²
0.1
0
0.3
Figure 3: The MAP estimator (lop) and the MMSE estimator (boltom) with respect to the obser- vation for e2 = 0.052 and different noise levels o².
Then the MMSE estimator is given by the expectation value of the posterior
AMMSE( (1) = arg min E P.x,)) - = =
xTx dx
p + 121712)
Wi + W2
y 20 1o cosh(
+ 0² + 2 sinh + 2))
y
11 + Wh E(E2 + σ2)
In Figure (bottom), we see that the MMSE estimator shows a smooth transition in particular for larger noise levels, mcaning that the estimator is robust against small perturbations the observation near zero. Note that case of Gaussian prior X (Nm,,)) Σ) in Rmm and white Gaussian noise, the MAP and MMSE estimators coincide.
B Local Lipschitz continuity of the generator for a latent space with infinite support
Here we show a weakened version of Lomma leading to an arbitrary small additive constant. The main difference is the weaker assumption || (Vy, =)|l < L, for all z E Rd with ||=|| S and all y E R" with |>|| <r, which fulfilled for continuously differentiable generators. For this we use the so-called truncated normal distribution (Horacc 2005 Tallis 1963). Let pz be the density of the
18

---

Published in Transactions Machine Learning Research (07/2023)
standard normal distribution Pz (N),, In), then the density of the truncated normal distribution Pžž is given by
pz(a)
p2(=),
if 1
(2) =
Ba(0) p2(=)d=
Cx
0,
else.
Lemma 10. Let Pz ~,)
In) be the latent space. For any parameterized fomily of generative models G with (v, 2)Il I Ly for all 2 E Rd with || S F and all E R with lllll S for some L, > 0 and some r > 0, it holds
y2ll + Mr W1 WG + Pz. G(ye, ) P2) 1r11
for all Vı, 32 R with ll 11 S r. The additive constant Mj fulfills Mp - 0 for F - oc. 00.
Proof. Let V1, I with |M lI M/ I1 S then it holds
Wi(G(yn P2 G(y2, )2 Pz) & Wi(G(y), ) Pz. G(y P%) + (G(())))) Pp)
(G ) Pž, G(22, ) Pz).
By the assumption on the generator G, Lemma !
yields
Wi(G(yn-) P2 G(yz, 4 P2) S 1r - 32ll-
Consequently, suffices to show that for y R with |g S the term Wi ( G(y,)) Pt) vanishes for - cc. By definition, it holds
((G(g22)dPP(()) Jfc +(G(g.)))dP(()
Wi(G(y Pz, Gly, )+P))
max Lip(F)
((G(y,)))dPP) (2((0)
p(G(u.z))dPz)
Lip(s)
BA
max Lip
((G 2))p2(2dd>
(L)
(GG(y, ((G(y,)))22(2)(C))))) J
C
(2
The first term vanishes exponentially in by the density pz: and for the second term note that C for f - co.
C
Experimental demonstration of assumption
Here we experimentally demonstrate that the expectation in gets small when training ditional generative network. For this, we consider a conditional normalizing flow as an example, consisting of three and ten Glow coupling blocks with fully connected subnets with one hidden
avaallable https://githhb com//VLL-HD/PrELL.
19

---

Published in Transactions Machine Learning Research (07/2023)
1
.... 10 blocks, 512 nodes
expected W1 distance
-3 blocks, 64 nodes
0.8
0.6
0.4
0.2
0
-2.5
-1.5
-1
-0.5
training loss
Figure 4: Expectation of the Wasserstein distance between posterior and pushforward of the gen- erator with respect to the training loss of the conditional normalizing flow.
layer and 64 and 512 nodes, respectively. As in Appendix A we choose two-dimensional Gaussian mixture model with six modes as prior distribution and additive Gaussian noise with standard deviation 0.5 for the noise model. The forward operator chosen to be the identity. Then we train the conditional normalizing flow for 100000 optimizer steps using Adam Kingma Ba (2015) with learning rale of le and batch size of 1024. Note that we can analytically compute the posterior distribution by Lemma, 9
In Figure f we visualize the expectation in 4 with respect to the training loss of the conditional normalizing flow, which is, up lo constant, equal to the KL, see Section 3.1 Obviously, the expectation (4) gets small when minimizing the training loss the conditional normalizing flow. We computed the Wasserstein distance using and draw 20000 samples from each distribution. Moreover, discretized the expectation by drawing 30 observations.
Pavailable
20


Reviews:
[{"review_id": "RDyiELDvRZ", "reviewer": "Reviewer_3wGv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I think the introduction is very well written, and the considered problem is interesting.\n\n- In section 3, the authors show that they can transfer their guarantees, which in Theorem 5 are for conditional generative models minimizing the 1-Wasserstein distance, to different generative models that minimize different statistical distances.\n\nWeaknesses:\n- The notation / writing is somewhat confusing:\n    - On the top of page 5, what is $G^{\\epsilon}$? \n    - Theorem 5 is very difficult to parse. The definition of $\\epsilon$ makes it hard to identify how small it is, and then the main bound in Eqn 5 is also difficult to parse.\n    - The authors keep using the example of additive Gaussian noise in the observations $y$, but the Lemmas, Theorems, and Assumptions are for $ \\| y \\| \\le r$, which wouldn't be satisfied with Gaussian noise. \n\n- The Lipschitz constant $L_r$ in Lemma 2 and the constant $C_r$ in Lemma 3 are treated as universal constants. But they may have a dimension dependence -- for example, if the generative model is a feedforward ReLU network, as $L_r$ is defined using $\\| \\nabla_y G(y, \\cdot) \\| \\le L_r$, this gradient can very well scale as $ c^{poly(m,n)}$, where $c$ is the max weight in $G$, and $n, m$ are the dimensions of $y, x$ (there will also be an exponential dependence on the number of layers in the generative model). This has not been considered in any of the results.\n\n- The strength of Theorem 5 is difficult to judge. Eqn 4 assumes that on average over $y$, the conditional generative model is close to the true posterior in Wasserstein distance (there are no algorithmic guarantees here, this is assumed to be true). Then, the authors show that for $y$ small enough (within a radius $k$, where $k$ depends on $a$, the marginal likelihood of $p_Y$ at the observed $y$), the Wasserstein distance between the posterior on this single sample $y$ is close to the true posterior distribution. It's dificult to judge how small these radiuses are (since $a$ can be exponential small for Gaussian distributions), how large the Lipshitz constants are, how small $\\epsilon$ is above Eqn 5 and as a result, how good a bound the RHS of Eqn 5 is.\n\n- I have some concerns about how well posed the problem is. The authors assume that in expectation over $y$, Eqn 4 gives $\\epsilon$ Wasserstein distance between the trained model and the true posterior distribution. Then they claim that the posterior distributions are close pointwise in $y$ -- this doesn't make sense since a single $y$ is a zero-measure set. Are the results almost surely over $y$? Are they with high probability over $y$? I think the authors need to be more precise about what exactly their probabilistic claims are.\n\n- Eqn 4 is assumed to be true for the population average over $y$, not just for a training set of $y$.\n\n- The analysis techniques are limited. The authors assume that the generative models have good Lipshitz constants, and don't discuss what models would have such Lipshitz constants. The results also end up being multiple triangle inequalities with Lipshitz inequalities employed strategically : to show that the conditional distribution at $y$ is close to the true distribution, the authors use the triangle inequality + symmetrization to make this bound between $y_1$ and $y_2$ that are bounded, then apply the Lipshitz inequality of $G$ between $y_1$ and $y_2$. This assumes that $y_1$ and $y_2$ lie in a small enough ball, and that the Lipshitz constant is small enough to make the error bound reasonable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the problem of learning posterior distributions in inverse problems. The authors show that assuming conditional generative models can minimize the Wasserstein distance to the true posterior distribution in expectation over the measurements, they can guarantee that the conditional generative model is close to the posterior distribution pointwise in the observations. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please state assumptions outside the Lemma statement, so that it's easier to identify what the assumptions are.\n\n- There needs to be more concentrated effort towards making the Theorem statements readable. It is very difficult to judge the quality of the bounds in their present form. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RAx3SFVQkl", "reviewer": "Reviewer_MHWT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is mostly well written and the theoretical results seem solid, though I haven't been able to carefully check all the proofs. The main issue to me seems to be the assumption of a well trained model satisfying the inequality in Eq. 4. I'm not sure how feasible it is to verify this condition in general for practical models. The applications section does not discuss, for example, how a small \n would be achievable for models of different complexities, as measured by, e.g., the number of layers in a conditional normalizing flow. It is, instead, based on the assumption that training reaches a reasonable approximation error level, which could be hard to verify as Eq. 4 assumes an expectation over the entire data distribution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper derives theoretical results on the stability of approximate posteriors for Bayesian inverse problems based on deep generative models with respect to perturbations in the observations. The results are based on the Lipschitz continuity of the generative models and its effects on the Wasserstein distance between the true posterior and the approximate one. Applications to popular deep conditional generative models are presented.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I acknowledge that there is already some discussion on the assumption in Eq. 4 in the paper. However, the section with the application of the theoretical results to me seems to need a more concrete demonstration that the assumptions are applicable to general models, maybe with an experimental demonstration, for example.\n\nThere are also a few minor issues: unusual quotation marks, quite a few important references are ArXiv preprints, such as Ardizzone et al. (2019), Bartzolis et al. (2021), etc., and in-text citations should use appropriate formatting, e.g., \"in Author (year)\", instead of \"in (Author, year)\".", "broader_impact_concerns": "N/A, paper of a theoretical nature", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y5I9za0Mhv", "reviewer": "Reviewer_SurZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a clear theoretical result that characterises the robustness of a broad class of approaches, effectively proving that some computationally appealing approaches used in general machine learning literature are applicable for solving Bayesian inverse problems. The result is nontrivial and relates intuitive concepts (Lipschitz constants and the training objective of the generator) to the upper bound on approximation error, and is likely to be useful both for practitioners and for further theoretical understanding of the the problem. The paper is in general well written and easy to understand for a theoretical paper and related work is covered well, and the claims are well justified by detailed theorems and proofs. There is no need for the paper to have empirical results of any kind.\n\nThe other main strength is detailed presentation of how the general result looks like in context of specific families of conditional generative models, covering effectively all of the main families currently used in the field. I find it highly useful that e.g. GANs and diffusion models are interpreted within the same framework, since this improves our understanding of their relationships more broadly, not just in the context of the specific setting (here Bayesian inverse problems). I commend the authors for taking the otherwise fairly general theorem to the level of practical application via these examples.\n\nThe only minor weaknesses concern the structure and presentation, detailed below:\n1. The structure and flow of Introduction is confusing. The first paragraph that talks about vulnerability of NNs and adversarial attacks is completely out of place here. Does it even belong in this paper? Even if the answer is yes, it definitely should not be the first paragraph, but you should directly start from the task of Bayesian inverse problems. Somewhat related to this, I do not see need for relating the work that much to 'end-to-end solutions' -- you are anyway doing the work in context of Bayesian inverse problems and can just start from stating that you are providing the Bayesian solution with the obvious advantages (as pointed out by numerous references) it has, and then proceed to explain how conditional generative models offer nice opportunities for the task but their theoretical properties are not yet known well enough.\n2. You refer in the main paper couple of times to the motivational example in Appendix. This is quite difficult for the reader since it requires jumping back and forth. If you think that the example is needed for clarifying the motivation, to the extent that you want to refer to it already in the second paragraph of the paper, then it should be in the main paper. My practical recommendation is to create a version that combines essential content of Figures 1 and 2 (for instance, one column of Fig 1 and 1-2 examples from Fig 2) and put it on the second page of the main paper. Then you can explicitly refer to that image and explain what we see there in Introduction. You can still leave the details of the model (Lemma 8 etc) in the Appendix as long as you just say in words in the main paper how this example corresponds to noisy linear forward operation for a mixture distribution. Alternatively, you could move the whole Appendix to become new Section 2 right after the Introduction, as an example that provides the reader the context needed for the theoretical development.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article characterises the robustness of broad range of conditional generative models in the task of Bayesian inverse problems. The paper provides a general theorem about pointwise robustness for a fairly general family of generative models, in form of Theorem 5 that provides an upper bound for Wasserstein distance between the true solution and the estimate in terms of smoothness of the inverse problem and the generator, as well as a threshold (again expressed in terms of the Wasserstein distance) for the quality of the generator. This general result is then related to several practical families of conditional generative models, explaining how normalizing flows, GANs, diffusion models, and VAEs can be analysed in this framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request the authors to:\n1. Carefully check the Introduction. Remove/move the first paragraph and try to streamline the story in the 2nd and 3rd paragraph. The core contribution of the paper is in analysis of how well some methods work for the Bayesian inverse problem and hence the core audience will be people already familiar with the reasons we solve that problem. Write for that audience.\n\n2. Bring the motivational example in Appendix A to the main paper. This can be done in many ways (some suggestions provided above), but I think something has to be done to make the paper work. The current approach where you refer to that example several times to motivate the work but still leave the whole description in Appendix is problematic in too many ways.", "broader_impact_concerns": "No need for discussing broader impact in this paper; it is obvious that improved theoretical understanding of the robustness of a computational approach is useful for improved transparency and quality of future research and bears no risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers are all in agreement leaning towards accepting this manuscript, they have not advocated for any certifications. I personally would recommend that the authors consider sharing their code in future submission.
Decision: Accept (poster)