Differentiable Gaussianization Layers for Inverse Problems Regularized by Deep Generative Models
Authors: Dongzhuo Li
Venue: ICLR 2023 poster (2023)
Keywords: Deep generative models
inverse problems
Gaussianization

Abstract: Deep generative models such as GANs, normalizing flows, and diffusion models are powerful regularizers for inverse problems. They exhibit great potential for helping reduce ill-posedness and attain high-quality results. However, the latent tensors of such deep generative models can fall out of the desired high-dimensional standard Gaussian distribution during inversion, particularly in the presence of data noise and inaccurate forward models, leading to low-fidelity solutions. To address this issue, we propose to reparameterize and Gaussianize the latent tensors using novel differentiable data-dependent layers wherein custom operators are defined by solving optimization problems. These proposed layers constrain inverse problems to obtain high-fidelity in-distribution solutions. We validate our technique on three inversion tasks: compressive-sensing MRI, image deblurring, and eikonal tomography (a nonlinear PDE-constrained inverse problem) using two representative deep generative models: StyleGAN2 and Glow. Our approach achieves state-of-the-art performance in terms of accuracy and consistency.

Published as a conference paper at ICLR 2023
DIFFERENTIABLE GAUSSIANIZATION LAYERS FOR INVERSE PROBLEMS REGULARIZED BY DEEP GENERA- TIVE MODELS
Dongzhuo Li
ExxonMobil Technology & Engineering Company dongzhuc. 1i@exxonmobil.. com
ABSTRACT
Deep generative models such GANs, normalizing flows, and diffusion models are powerful regularizers for inverse problems. They exhibit great potential for helping reduce ill-posedness and attain high-quality results. However, the latent tensors of such deep generative models can fall out of the desired high-dimensional standard Gaussian distribution during inversion, particularly in the presence of data noise and inaccurate forward models, leading to low-fidelity solutions. To address this issue, we propose to reparameterize and Gaussianize the latent tensors using novel differentiable data-dependent layers wherein custom operators are defined by solving optimization problems. These proposed layers constrain inverse problems to obtain high-fidelity in-distribution solutions. We validate technique on three inversion tasks: compressiveseensing MRI, image deblurring, and eikonal tomogra- phy (a nonlinear PDE-constrained inverse problem) using two representative deep generative models: StyleGAN2 and Glow. Our approach achieves state-of-the-we performance in terms of accuracy and consistency.
INTRODUCTION
Inverse problems play a crucial role in many scientific fields and everyday applications. For example. astrophysicists use radio electromagnetic data to image galaxies and black holes (Högbom 1974
Akiyama et al., (011)) Geoscientists rely seismic recordings to reveal the internal structures of Earth (Tarantola a 1984; Tromp et al.) 2005; Virieux & Operto 2009). Biomedical engineers and doctors use X-ray projections, ultrasound measurements, and magnetic resonance data to reconstruct images human tissues and organs (Lauterbur) 1973; Gemmeke & Ruiter 2007] Lustig et al.) 2007). Therefore, developing effective solutions for inverse problems of great importance in ad advancing scientific endeavors and improving our daily lives.
Solving an inverse problem starts with the definition of forward mapping from parameters to data d, which we formally write as
d= fm) +
where f stands for forward model that usually describes some physical process, e denotes noise, the observed data, and m the parameters to be estimated. The forward model can be either linear or nonlinear and either explicit or implicitly defined by solving partial differential equations (PDEs). This study considers three representative inverse problems: Compressive Sensing MRI, Debturring, and Eikonal (traveltime) Tomography, which have important applications in medical science, geoscience, and astronomy. The details of each problem and its forward model are in App.
The forward problem maps m to d, while the inverse problem estimates m given Unfortunately, inverse problems are generally under-determined with infinitely many compatible solutions and intrinsically ill-posed because of the nature the physical system. Worse still, the observed data are usually noisy, and the assumed forward model might be inaccurate, exacerbating the ill- iol-posedness. These challenges require using regularization to inject a priori knowledge into inversion processes obtain plausible and high-fidelity results. Therefore, an inverse problem is usually posed as an optimization problem:
arg min (1/2) | m
(m) R(m),
(2)

---

Published as a conference paper at ICLR 2023
Latent Lensor properties
Output Before correction
Output Alter correction
Skewed
Corrected
(a)
YJ layer
9.4
DGM
0.2
0.0
Heavy-tailed
Corrected
0.6
(b)
DCM
0.4
LB layer
02
0.0
1000
0.05
Non-i.idd
Pattern destroyed
(c)
DGM
Full G layers
DCM
Figure 1: Comparison of images generated by deep generative model (DGM), Glow, using latent tensors that deviate from a spherical Gaussian distribution (left) and those after corresponding corrections (right). The visual effects highlight the necessity of keeping the latent tensor within such distribution during inversion. The second column shows the characteristics deviated latent tensors: (a) histogram: i.i.d. components but the distribution is skewed; (b) histogram: i.i.d. components but the distribution is heavy-tailed; (c) latent tensor image: non-i.i.d. entries. The first column shows the corresponding outputs of Glow network. The third column shows latent tensors corrected by (a) the Yeo-Johnson layer (YJ), (b) the Lambert W x Fx layer (LB), and (c) the full set of our Gaussianization layers (G layers). Those corrected latent tensors map to realistic images shown in the fourth column. All latent tensors have norm of 0.7 / dim because of the Gaussian Annulus Theorem (App. hp and the fact that Glow works best with a temperature smaller than one (see Fig. 2 Additional examples for StyleGAN2 and Stable Diffusion (Rombach et al.) 2022 can be found in App.
where R(m) is the regularization term. Beyond traditional regularization methods such as Tikhonov regularization and Total Variation (TV) regularization, deep generative models (DGM), such
as VAEs (Kingma& & Welling 2013), GANs (Goodfellow et al.) (2014), and normalizing flows (Dinh)
2014|[206]] Kingma al. [016)] Papamakarios et al. 2017 Marineseu al., 2020). have shown
great potential for regulan aegularizing inverse problems (Bora et al. [017]] Van Veen et al. (0018) Hand et al.
2018 Ongie et al. 2020 Asim et 2020 Mosser et al. 22200] Li al. 2021 Siahkoohi et al. |
Whang et al. 2022 1 Cheng et al. [2022] Daras et al. 2021 2022). Such deep generative models directly
learn from training data distributions and are a powerful and versatile prior. They map latent vectors
z to outputs m distributed according to an priori distribution: m g(z) Pearget, ~ (O,
for example. The framework of WGM-regularized inversion (Bora al. 2017
arg min (1/2) ||d y (2)1l² R')2),
where the deep generative model whose layers are frozen, reparameterizes the original variable m, acting as a hard constraint. Instead of optimizing for m, we now estimate the latent variable and retrieve the inverted m by forward mappings. Since the latent distribution is usually a standard Gaussian, the new (optional) regularization term R' (z) can be chosen as |||| for GANs and VAEs, where I is weighting factor. See App. J.I for more details flows. Since the optimal 3 depends on the problem and data, tuning 3 is highly subjective and costly.
similar formulation for normalizing
However, this formulation of DGM-regularized inversion still leads to unsatisfactory results if the data are noisy or the forward model is inaccurate, as shown in Fig. 20 even if we fine-tune the weighting parameter B.
To analyze this problem, first recall that a well-trained DGM has a latent space (usually) defined on standard Gaussian distribution. In other words, DGM either only sees standard Gaussian latent
2

---

Published as a conference paper at ICLR 2023
tensors during training (e.g., GANs) learns to establish 1-1 mappings between training examples and typical samples from the standard Gaussian distribution (e.g., normalizing flows, App. (.2). A.2. As a result, the generator may map -iittii-ittiittiinnn latent vectors unrealistic results. We show in Fig. the visual effects of several types of deviations of latent vectors from a spherical Gaussian (with temperature of 0.7). It can be seen that (1) the latent tensor should have i.i.d. entries, and (2) these entries should be distributed as ID standard Gaussian in order to generate plausible images. Since the traditional DGM-regularized inversion lacks such Gaussianity constraint, we conjecture that the latent tensor deviates from the desired high- high-dimensional standard Gaussian distribution during inversion, leading to poor results. Our observations and reasoning motivate us to propose set of differentiable Gaussianization layers to reparameterize and Gaussianize the latent vectors of deep generative models (e.g., StyleGAN2 (Karras et al.) 2020) and Glow (Kingma & Dhariwal, 2018)) for inverse problems. The implementation is available here
2 METHOD
2.1
OVERVIEW REPARAMETERIIZTION USING GAUSSIANIZATION LAYERS
Our solution is based on necessary condition of the standard Gaussian prior latent tensors. Let us define a partition of latent tensor E In as the collection of non-overlapping patches P(z) i=l... ii.,.,,,,, ,N. where the patches z; RR are of the same dimension and can be assembled as 2, Le., n n= N x D. If the latent tensor z ~ N(, I), then for all from any partition P(z), we have z, ~ ().
Note that z is a symbolic representation of the latent tensor. In a specific DGM, can be either 2D/3D tensor or list of such tensors corresponding to multi-scale architecture (App. D). Even though there are numerous partition schemes, such as random grouping of components, we choose the simplest: partitioning z based on neighboring components, which works well in practice.
To constrain
by constructing a mapping h: V, - 2, such that z; ~ (0, The new variables v, are of the same dimension as z, Suppose that we have constructed the patch-level mapping h, we can obtain mapping ht at the tensor level, such that z = ht (v), where v is assembled from V; in the same way as z from Z, For example, ht diag(h..... h) if patches are extracted from neighboring components and concatenated into a vectorized v. Hence, the original DGM-regularized inversion 3 becomes
10, during inversion, reparameterize
arg min mrg
(1/2) ||d gogo ht (v)lle-
(4)
Since we are imposing a constraint through reparameterization, there is no need to include the regularization term R'zz). The new formulation is still an unconstrained optimization problem, enabling us use highly efficient unconstrained optimizers, such as L-BFGS (Nocedal & Wright 2006) and ADAM (Kingma & Ba 2015).
The remaining critical piece is construct h, and leads to our Gaussiamization layers. First, we translate the constraint of z, = h(vv)) ~ (O, I) into the following optimization problem:
arg min DKL. (pz (h (v:)) ())...))
(5)
where pz is the probability density function (PDF) of z,. Second, we adopt the framework proposed by precursor works of normalizing flows Gaussianization (Chen & Gopinath 2000 Laparra et al. 2011) to solve this optimization problem. The KL-divergence can be decomposed as the sum of the
inltiiinformatinn [2,, and the marginal negentropy (z,) (Chen & Gopinath 2000 Meng et al.
2020
DKL. (pz (z:) II, (0,1)) I(z:) + Jm(Zz),
(6)
where
|[ and Jmn (z.) = Σ DEL
D
T(2,)= = DRL.
Pz (z:)
tests are challenging for high- dimensional data RRabanser et al. Nalisnick et al.] (2019). Also, since latent tensors are Graussan/Gauussan-iike it is hard to conduct dimension reduction for such tests.
2019
3

---

Published as a conference paper at ICLR 2023
Here 2,0) denotes the j-th component of patch vector 2;, and pj stands for the marginal PDF for that component. The multi-information [z;) quantifies the independence of the components of Z;, while the marginal negentropy Jm (z,) describes how close each component is a ID standard Gaussian. With this decomposition, the optimization procedure depends on the facts: (1) the KL divergence and a standard Gaussian in 6q. are invariant to an orthogonal transformation, and (2) the multi-information term is invariant to a component-wise invertible differentiable transformation (App. J.3). As a result, we perform Gaussianization in two steps:
1. Minimize the multi-information I [z.). This is done by an orthogonal transformation that keeps the overall KL, divergence the same but increases the negentropy Jm (2:). We achieve this by using our independent component analysis (ICA) layer. ICA is the optimal choice since it maximizes non-Gaussianity so that the subsequent marginal Gaussianization step removes and results in a large decrease in DEL (pz (h (v:)) (0, I)).
2. Minimize the marginal negentropy (z.) by component-wise operations that perform ID Gaussianzzation of marginal distributions p=. The multi-information does not change under component-win -iise invertible operations. Therefore, the overall KL divergence between Z; and the Gaussian distribution decreases.
The Gaussianization steps are well-aligned with the motivating example (Fig. D To constrain DGM outputs be plausible, should make components within latent tensor patches independent (or destroy the patterns) (Fig. [c) and shape the ID distribution as close as possible to Gaussian (Fig. (b)).
We will see that h is parameterized by an orthogonal matrix and two scalar parameters in ID Gaussian- ization layers. Unlike conventional neural network layers, the input-data-depende Gaussianization layers are not defined by learning from a dataset ahead of time but by solving certain optimization problems on the job (Fig. T8] Special care should be taken to implement the gradient computation correctly and ensure that they pass the mifte-difference convergence test (App. G As an overview, the composition of our proposed layers is:
- Whitening -> ICA Yeo-Johnson - Lambert W Fx Standardization +
where whitening and ICA belong to the first step and the rest belong to the second step. We will discuss in App. K some possible simplifications of the layers in practice after our ablation studies.
The overall proposed inversion process (one iteration) is illustrated in Fig.
G layers h1
DGM
Forward
Compute mislit
z
m
d
Defined
model
Calculated data
Obseeved data
Gradient- update-
Illustration of the proposed inversion process. Gradient computation in the Gaussianization layers is enabled by the implicit function theorem and automatic differentiation (App. G We use the L-BFGS optimizer (Nocedal & Wright 2006) update
Figure 2:
2.2 REDUCING MULTI-INFORMATIION ICA LAYER
The orthogonal matrix W is constructed by the independent component analysis (ICA). The input is the patch vectors ( vvli=i,....,v. The ICA algorithm first computes the input-dependent orthogonal matrix W and then computes p; = W Viriml,...,, as the output. The orthogonal matrix W makes the entries of each p; independent random variables.
Hyvärinen & Oja 2000), which employs a fixed- point algorithm maximize contrast function + (e.g., the logcosh function), for our ICA layer to reduce multi-information.
We use the FastICA algorithm (Hyvarinen) 1999
The FastICA algorithm typically requires that the data are wre-white We adopt the ZCA whitening method or the iterative whitening method introduced in Hyvarinen] (1999) (App. With the whitened data, we compute W using a damped exentpoint iteration scheme:
1 W = N
a (VV
W (o' 1)
(8)
4

---

Published as a conference paper at ICLR 2023
where 1 is all-one vector, the column vectors of V are {vale=1,....N1 (..)))) (')) (0, 1). and we use a = 0.8 throughout our experiments. To save computation time, we only perform maximum of 10 iterations. The details of the whole algorithm can be found in App. E
We set the initial W as an identity matrix. If the imput vectors are already standard Gaussian (in- distribution), the computed W will still be identity matrix, which maps the input the same output. In practice, the empirical distribution from finite samples is not a standard Gaussian, so W not an identity matrix but another orthogonal matrix, which still maps standard Gaussian input vectors to standard Gaussian vectors as output. For this reason, we sample starting [Vilimt,....N ,N from the standard Gaussian distribution start inversions with plausible outputs.
2.3 REDUCING MARGINAL NEGENTROPY
For ID Gaussianization, we choose a combination of the Yeo-Johnson transformation that reduces skewness and the Lambert Fx transformation that reduces heavy-tailedness. Both are layers based on optimization problems with only one parameter, which is cheap to compute and is easy to back-propss ppoo-proe the gradient. Eq. 7 requires us to perform such ID transformations for each component of the random vectors. In other words, we need to solve the same optimization problem for D times, which imposes a substantial computational burden. Instead, we empirically find acceptable to share the same seanmiiiaatoonggnnratttt parameter actoss all components. In other words, we perform only a single ID Gaussianization, treating all entry values the latent vector as the data simultaneously.
Power transformation layer We propose to use the power transformation or Yco-Johnson trans- formation Yeo & Johnson| 2000) to reduce the skewness of distributions. As shown in Fig. B(a), the form of the Yeo-Johnson activation function depends on parameter X. If X = 1, the mapping > 1, the activation function convex, compressing the left tail and extending the right tail, reducing the left-skewness. If X 1, the activation function is concave, which oppositely reduces the right-skewness. We refer readers to App. for details.
an identity mapping.
Lambert W Fx layer Due to noise and inac- curate forward models, we observe that the distri- bution of latent vector values tends be shaped as a heavy-tailed distribution during the inversion process. To reduce the heavy-tailednnes, we adopt the Lambert W x Fx method detailed in Goerg (2015).
(a)
(b)
10
We use the parameterized Lambert W x Fx distri- bution family approximate a heavy-tailed input and solve an optimization to estimate optimal parameter (App. E). with which the inverse trans- formation maps the heavy-tailed distribution to- wards Gaussian.
Figure 3: The nonlinear activation functions
from (a) the power transformation (Yeo-Johnson) layer and (b) the Lambert W x Fx layer.
Fig. b)) shows that the Lambert W x Fx layer acts nonlinear squashing function. As d increases, it compresses more the large values and reduces the heavy-tailedness. Intuitively, the Lambert W Fx layer also be interpreted an intelligent way of imposing constraints on the range of values instead of simple box constraint. We refer the readers to App. for more details about the optimization problem and implementation.
Standardization with temperature Since the Lambert W x Fx layer output may not necessarily have a zero mean and unit (or prescribed) variance, we standardize the output using
z = (x E[<=)// Var(x)
(9)
where y is the temperature parameter suggested in Kingma & Dhariwal 2018
5

---

Published as a conference paper at ICLR 2023
3 RELATED WORK
End-to-end NNs for inverse problems for inverse problems, using CNNs (Chen et 2017 Jin et al. 2017 Sriirm et al. (020). GANs (Mar) dani et al. 2018: Lugmayr et al., 2020, Wang et al. 2018), invertible networks (Ardizzone et al. 2018), and diftusion models (Kawar et 2022). The general idea is simple: train a neural network that directly maps observed data to estimated parameters. Even though such methods seem effective in a few applications, DGM- regullrrre inversion with our Gaussianization layers is preferable for the following reasons. First, the forward modeling can be so expensive computationally that it is infeasible to collect a decent training datasets for some applications. For example, one large-scale fluid mechanics or wave propagation simulation can take hours, if not days. Second, the relationship between parameters and data can be highly nonlinear, and multiple solutions may exist. An end-to-end network may map data interpolated solutions that are not realistic. In comparison, our method can start from different initializations or even employ sampling techniques to address this issue. Third, the configuration of data collection can change from experiment to experiment. It is impractical, for example, to re-train the network each time we change the number and locations of sensors. In contrast, not only can our method deal with this situation, but it can even use the same DGM for different forward models, as can see in the compressive sensing MRI and cikonal tomography examples. While almost all end-to-end methods are only applied to linear inverse problems, our Gaussianization layers are also effective in nonlinear problems.
There are numerous end-to-end neural networks designed
inversion In high digeesiiensionall space, the proba- bility mass of a standard Gaussian distribution concentrates within the so-called Gaussian typical set (App. D.. To be in the Gaussian typical set, one necessary but not sufficient condition is to be within an annulus area around nigh-dimensional sphere (App. [I). Utilizing this geometric property, DGM- regularized inversion methods like [Bojanowski et al. (2117) and [Lang et al. (2021) ) force updated latent vectors to stay on the sphere. This strategy is closely related to spherical interpolation (White) 2016). We call this strategy the spherical constraint for inversion. In the original StyleGAN2 paper. the authors also noticed that in image projection tasks, the noise maps tend to have leakage from signals (Karras et al., 2020) the same phenomenon we discussed. They proposed a multi-scale noise regularization term (NoiseRlg) to penalize spatial correlation in noise maps. We extend the same technique to our inverse problems for comparison. Note that we use whitening layer before the ICA layer. The whitening layer can be used alone, similar to Huang et al. (2018) and Siarohin ct al. 2018), whose performance will be reported in ablation studies. Also, Wulff & Torralba (2020) observed that for StyleGAN2, the Leaky ReLU function can "Gaussianize" latent vectors in the W
Other techniques to improve
space. CSGM-w (Kelkar & Anastasio 202I) utilizes this a priori knowledge to improve DGM-
regularized compressive sensing problems. To further compare with the Gaussianizaion layers, we in
addition propose an alternative idea that reparameterizes latent vectors using learnable orthogonal matrices (Cayley parameterization) and fixed latent vectors, which closely related the work of orthogonal over- parameterized training (Liu et al. 2021) (App. H ). Recently, score-based generative models have started to show promise for sol solving inverse problems (Jalal et al., 202T Song et al. 2021). However, they have been mainly applied to linear inverse problems and challenged by noisy data (Kawar et al.) (20TT). Besides, scored-based methods might not work for certain physics-based inverse problems, since Gaussian noise parameters may break the physics simulation engine?
EXPERIMENTS
We consider three representative inversion problems for testing: compressive sensing MRI, image deblurring, and eikonal traveltime tomography. For MRI and eikonal tomography, we used synthetic brain images as inversion targets and used the pre-trained StyleGAN2 weights from Kelkar & Anastasio (2021) (trained on data from the databases of fastMRI (Zbontar et al. 2018 Knoll et al. 2020). FCIA-GBM (Scarpace et al. (016), and OASIS-3 (LaMontagne et al. 2019) for regularization. We used the test split of the CelebA-HQ dataset (Karras al. (2018) for deblurring, and the DGM is a Glow network trained on the training split. We refer readers to App. F for details on datasets and training.
For example, in elastic waveform inversion, initializing material properties using Gaussian noise may create unrealistic scenarios where the P-wave speed is lower than the S-wave speed at some spatial locations.

---

Published as a conference paper at ICLR 2023
We tested each parameter configuration in each inversion 100 images (25 in the eikonal tomography due to its expensive forward modeling). Since the deep generative models are highly nonlinear, the results may get stuck in local minima. Thus, we started inversion using three different randomly initialized latent tensors for each of the 100 or 25 images, picked the best value among the three for each metric, and reported the mean and standard deviation of those metries, except for CSGM-w. TV, and NoiseRig, where the initialization is fixed. The metrics we used were PSNR, SSIM (Wang al. 2004), and an additional LPIPS (Zhang et al. 2018) for the CelebA-HQ data. We used the LBFGS (Nocedal & Wright, 2006) optimizer in all experiments except TV, noise regularization, and CSGM-w, which use FISTA (Beck & Teboulle] 2009) or ADAM (Kingma & Ba 2013). The temperature was set to 1.0 for StyleGAN2 and 0.7 for Glow.
4.1 COMPRESSIVE SENSING MRI USING STYLEGAN2
The mathematical model of compressive sensing MRI is
d Am + e,
(10)
where A E CM is the sensing matrix, which consists of FFT and subsampling in the k-space (frequency domain). Eq. i under-determined system, and we use Accl N/M to denote the acceleration ratio. We also added i.i.d. complex Gaussian noise with a signal-to-noie ratio (SNR) of 20 dB 10 dB to the measured data. See [......ofor more background information.
Table I compares the results from total variation regularization (TV), noise regularization (NoiseRIg) Karras et al. 2020), spherical constrain/reppaaaeeeriizioo 2 = = * / dim(v),
CSGM-w (Kelkar & Anastasio 2021), our proposed orthogonal reparameterization (Orthogonal), and our proposed Gaussianization layers (G layers). Fig. 4 shows examples of inversion results. In the base case where AccI=8x and SNR=20 dB, the Gaussianization layers gives the best scores, and this advantage gets more significant when data SNR decreases to 10 dB. Interestingly, the scores from all methods improve significantly if we make the system better determined (i.e., Accl=2x), and the performance of TV, spherical constraint, and Gaussianization layers become more similar in this scenario. We conclude that our proposed Gaussianization layers are effective and more robust than other methods, particularly in low-SNR scenarios.
True
TV
Spherical
CSGM-w
NoiseRIg
Orthogonal
G layers
PLR.I
Figure 4: Comparison of compressive sensing MRI inversion results (Acel=8x, SNR=20 dB).
Ablation study Table summarizes the ablation study on the components of the Gaussianization layers. We kept the standardization layer on for all cases. The conclusions are as follows: 1. The ICA layer plays the most significant role in improving result scores; 2. The whitening/ZCA layer alone is not effective; 3. The Yeo-Johnson (YJ) and the Lambert W x Fx (Lambt) layers are more effective when the noise level is higher (e.g., SNR=10 dB vs. 20 dB). Their performance seems data-dependent: when SNR=20 dB, YJ seems more effective, while Lambt gives the best scores when SNR=10 dB. But the overall improvement from these two layers is marginal. In practice, one may only use ICA and one of the ID Gaussianization layers. Additionally, we tested the effect of patch size on the z (style) vectors (App. C). We find that the largest possible patch size gives the best results.
7

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of compressive sensing MRI results.
Acel 8x. SNR 20 dB
Accl 8x. SNR 10 dB
Acel= 2x. SNR 20 dB
Method
PSNR
SSIM
PSNRT
SSIM
PSNRT
SSIM
TV
1..
0.60
1 1.32
0,57 10.058
3.39
0.91 1:0024
Spherical
26.93+t657
a.79
22.78:
0.61 0.61 1:0.46
32.90+3.50
0.90
CSGM-w
20.19
0.57
19.98
0.55
27.53
0.74s
NoiseRgl
21.61:22.2
0.50
1..0
0.27:10.006
28.04
0.66:10007
Orthogonal
26. 14
0.784
25. 50
0.77:10.332
29.29
0.84
G layers
45.9
0.83++a.12
25.48 44.76
+.78
32.41 +-400
0.90+
Table 2: Ablation study of the Gaussianization layers.
Acel 8x. SNR 20 dB
Acel = 8x. SNR
10 dB
Method
PSNRT
SSIMT
PSNR1
SSIM1
ICA (X),
YJ (X),
Lambt (X)
26.932640
0.787
22.98=1.54
0.6042n252
ICA (X).
YJ (v).
Lambt (X)
26.92:46.42
4.717
23.14
0.622.40.45
ICA (X).
YJ (X).
Lambt (V)
25.33
0.743
23.58
0.695
ZCA,
YJ (X),
Lambt (X)
27.08::652
0.786=t0157
22.94
0.623
ICA (V),
YJ (X),
Lambt (X)
27.91
0.824
25.22=476
0.770
ICA (V),
(V), Lambt (X)
27.99
0.831 1:0128
25.48
0.779
ICA (V),
(X), Lambt (V)
27.21
0.816
24.57
0.779
ICA (V), YJ
Lambt (V)
27.37:5.90
0.81920135
25.09
0.771
4.2 IMAGE DEBLURRING USING GL.OW
The mathematical model for deblurring is
H=
(11)
where H is smoothing filter and denotes convolution. We used Gaussian smoothing filter with standard deviation 3, and added noise e ~ (N,, to the observed data. Although the system may not be under-determined, the high-frequency information is lost due to low-pass filtering: hence this is also an ill-posed problem. Though we only tested on facial images, deblurring has wide applications in astronomy and geophysics. See App. A.2 for more background information.
Original
Observed
Spherical
Orthogonal
Original
Observed
Spherical
Orthogonal
G layers
Figure 5: Comparison of deblurring results from different methods.
Table Jnd Fig. show that the Gaussianization layers are also effective in Glow, better than using the spherical constraint or the orthogonal reparameterization. We also demonstrate the efficacy of Gaussianization layers when the forward model is inaccurate, in which case the induced error in data not Gaussian (App. C
Ablation study Table 3 also shows that the two parameterization schemes (App. D for Glow using the Gaussianization layers have similar performance. Besides, we report the ablation study on the components of the Gaussianizaiton layers for Glow (App. C).
8

---

Published as a conference paper at ICLR 2023
4.3 EIKONAL TOMOGRAPHY USING STYLEGAN2
In acoustic wave imaging (e.g., ultrasound tomography), we excite waves using sparsely distributed sources one time at the boundary of the object. Then we reconstruct its internal structures (the spatial distribution of wave speed) given the first-arrival travel time recorded on the boundary. The following eikonal equation approximately describes the shortest travel time T(x; x:.) that the acoustic wave emerging from the source location I, takes to reach location inside the target object (Yilmaz 2001:
VT (z; a.,) = (a), T( x.)) = 0.
(12)
where <(x) is the wave propagation speed at each location. Both this eikonal PDE and the implicitly defined forward mapping c(x) + T(x) are nonlinear, and there has been little research on DGM-
regularized inverse problems with such nonlinear characteristics. The inverse problem is severely ill- posed, which is equivalent to curved-ray tomography problem. See AA.3 for more background information. We added noise to the recorded travel time using the following formula: Tooisy (at,; at,) = Tæx,; xx) (1+ + e). where ~ (O, 0.0012) and æ, denotes any receiver location. In other words, longer traveltime corresponds to larger uncertainties.
We show that the Gaussianization layers outperformed other methods in this tomography task in Table 4 and Fig. 6 Note that this is statistical conclusion. We also report an example where the spherical constraint works better than the Gaussianization layers (bottom right).
Table 3: Deblurring results using Glow.
Table 4: Eikonal tomography using StyleGAN2
Method
LPIPS
PSNR1
SSIM
Method
PSNRT
SSIM
Spherical
0.17+006
21.78
0.580
TV
20.66+1.1 0.543=0.733
Orthogonal
0.161006
21.91
0.583
Spherical
22. 19:405 0.686=0.144
G layers P1
22.40 0. 0.13
0.583
G layers
24.80 0.803=m.093
G layers P2
0. 0.13
22.47
0.590
True
TV
Spherical
G layers
True
TV
Spherical
G layers
PAPO3 Sone s0
Pants st0
Sapp 2349.99
SAP09 50 S0 M0
PAROs samie
Pub Joch st
Figure 6: Comparison of eikonal tomography results from different methods.
5 DISCUSSION AND CONCLUSIONS
We provide insights on the experiment results and further discuss this work's limitations, computa- tional cost, and broader impact in App. K
In summary, we have identified a critical problem in DGM-regularized inversion: the latent tensor can deviate from a typical example from the desired nigh-dimensional standard Gaussian distribu- tion, leading to unsatisfactory inversion results. To address the problem, we have introduced the differentiable Gaussianization layers that reparameterize and Gaussianize latent tensors so that the inversion results remain plausible. In general, our method has achieved the best scores in terms of mean values and standard deviations compared with other methods, demonstrating our method's advantages and high performance in terms of accuracy and consistency. (Regarding the standard deviation of scores, we only compare with other methods with competitive mean scores, such as orthogonal reparameterization and spherical constraint.) Our proposed layers are plug-and -la--lly.. require minimal parameter tuning, and can be applied to various deep generative models and inverse problems.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The author would like to thank Huseyin Denli, Ashutosh Tewari, Myun-Seok Cheon, Di Du, Stuart Harwood, Yu Fan, and Qiuzi Li for helpful discussions and the anonymous reviewers for their constructive feedback, which greatly improved the paper.
REFERENCES
Kazunori Akiyama, Antxon Alberdi, Walter Alef, Keiichi Asada, Rebecca Azulay, Anne-Kathrin Baczko, David Ball, Mislav Baloković, John Barrett, Dan Bintley, et al. First m87 event horizon telescope results. iv. imaging the central supermassive black hole. The Astrophysical Journal Letters, 8,
Lynton Ardizzone, Jakob Kruse, Sebastian Wirkert, Daniel Rahner, Eric W Pellegrini, Ralf S Klessen, Lena Maier-Hein, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv: 04730, 2018.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 399-409. PMLR, 2020.
Amir Beck and Marc Teboulle. A fast iterative taankkgg--threshodingg algorithm for linear inverse problems, SIAM journal on imaging sciences, 2(1):183-202; 2009.
Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Cambridge University Press, 2020.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint (rXiv:1707.0570 2017.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using generative models. In International Conference Machine Learning, pp. 537-546. PMLR, 2017.
Richard P Brent. Algorithms for minimization without derivatives. Courier Corporation, 2013.
Hu Chen, Yi Zhang, Mannudeep K Kalra, Feng Lin, Yang Chen, Peixi L Liao, Jiliu Zhou, and Ge Wang. Low-dose et with residual encoder-decoder convolutional neural network. IEEE transactions on medical imaging, (1111)))2224-5355 2017.
Scott Chen and Ramesh Gopinath. Gaussianization. Advances in neural information processing systems, 13:423-429, 2000.
Yen-Chi Cheng, Chieh Hubert Lin, Hsin- Ying Lee, Jian Ren, Sergey Tulyakov, and Ming-Hsuma Yang. Inout: Diverse image outpainting via gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 11431-11440, 2022.
Robert M Corless, Gaston H Gonnet, David EG Hare, David Jeffrey, and Donald E Knuth. On the lambertw function. Advances in Computational mathematics, 5(1):329-359 1996.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. Wiley, 2012. ISBN 7781118585771 URL httttt:tt//ttttto google wwwo//oookkk http:/oooook.oooossssssoooooo.aa
Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros Dimakis. Intermediate layer optimization for inverse problems using deep generative models. arXiv preprini arXiv:2102.07364 2021.
Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. Score-guided intermediate layer optimization: Fast langevin mixing for inverse problem. arXiv preprins arXiv:2206.09 104, 2022.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14Xiv:1410.85 16, 2014.
Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605v 08803, 2016.
10

---

Published as a conference paper at ICLR 2023
H Gemmeke and NV Ruiter. 3d ultrasound computer tomography for medical imaging. Nuclear Instruments and Methods Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 58002):057--1655, 2007.
Georg M Goerg. The lambert way to gaussianize heavy-tailed data with the inverse of tukey's 'sh transformation as special case. The Scientific World Journal, 2015, 2015.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.26611 2014.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3012-3021, 2020.
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retrieval under generative prior. arXiv preprint /80i 00221,04,11111 2018.
JA Högbom. Aperture synthesis with a non-regular distribution of interferometer baselines. Astronomy and Astrophysics Supplement Series, 15:417, 1974.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 791-800, 2018.
Aapo Hyvarinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE transactions on Neural Networks, 10(3):626-6) 634, 1999.
Aapo Hyvärinen. The fixed-point algorithm and maximum likelihood estimation for independent component analysis. Neural Processing Letters, 10(1):1-5, 1999.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, ((4-5):411-430 2000.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34: 1ee114498-99545 2021.
Kyong Hwan Jin, Michael T' McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems imaging. IEEE Transactions on Image Processing, 26(9): 4509-4522, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. 2018.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 8110-8119, 2020.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochas- tically. Advances in Neural Information Processing Systems, 34:21757-21769 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201. 2022.
Varun Kelkar and Mark Anastasio. Prior mangecconttrained reconstruction using style-based generative models. In International Conference on Machine Learning, pp. 5367-5377. PMLR, 2021.
Diederik Kingma and Jimmy Ba. Adam, method for stochastic optimization. 2015.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. arXiv preprint arXiv: 1807,03039 2018.
Diederik Kingma and Max Welling. arXiv:1312.60 2013.
Auto-encoding variational bayes.
arXiv preprint
11

---

Published as a conference paper at ICLR 2023
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint arXiv:1606.048 04934, 2016.
Florian Knoll, Jure Zbontar, Anuroop Sriram, Matthew J Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Krzysztof J Geras, Joe Katsnelson, Hersh Chandarana, et al. fastmri: A publicly available k-space and dicom dataset of knee images for accelerated image reconstruction using machine learning, Radiology: Artificial Intelligence, 2(1): 2020.
Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Horbeeck Chengjie Xiong. Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, al. Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRxiv, 2019.
Valero Laparra, Gustavo Camps-Valle and Jesús Malo. Iterative gaussianization: from ica random rotations. IEEE transactions neural networks, 22(4):537-549, 2011.
Paul C Lauterbur. Image formation by induced local interactions: examples employing nuclear magnetic resonance. nature, 242(5394):190-191, 1973.
Dongzhuo Li, Kailai Xu, Jerry M Harris, and Eric Darve. Coupled time-lapse full-waveform inversion for subsurface flow problems using intrusive automatic differentiation. Water Resources Research, 56(8):e2019) 0A0.0019RRRR203300 2020.
Dongzhuo Li, Huseyin Denli, Cody MacDonald, Kyle Basler-Reeder, Anatoly Baumstein, and Jacquelyn Daves. Multiparameter geophysical reservoir characterization augmented by generative networks. In First International Meeting for Applied Geoscience & Energy, pp. 1364-1368. Society of Exploration Geophysicists, 2021.
Jingyun Liang, Kai Zhang, Shuhang Gu, Lue Van Gool, and Radu Timofte. Flow-based kernel prior with application to blind super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10601-10610, 2021.
LR Lines and S Treitel. A review of least-squares inversion and its application to geophysical problems. Geophysical prospecting, 32(2):159-186, 1984.
Weiyang Liu, Rongmei Lin, Zhen Liu, James M Rehg. Liam Paull, Li Xiong, Le Song, and Adrian Weller. Orthogonal over-parameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7251-7266 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super-resolution space with normalizing flow. In European Conference on Computer Vision, pp- 715-732. Springer, 2020.
Michael Lustig, David Donoho, and John M Pauly. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 58(6): 182-1199.11191 2007.
Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas S Vasanawala, Greg Zaharchuk, Lei Xing. and John M Pauly. Deep generative adversarial neural networks for compressive sensing mri. JEEE (ransactions on medical imaging, 38(1) ): 167--19, 2018.
Razvan V Marinescu, Daniel Moyer, and Polina Golland. Bayesian image reconstruction using deep generative models. arXiv preprint 20X1v 2020.
Chenlin Meng, Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization flows. In International Conference Artificial Intelligence and Statistics, pp. 4336-4345. PMLR, 2020.
Lukas Mosser, Olivier Dubrule, and Martin J Blunt. Stochastic seismic waveform inversion using generative adversarial networks as a geological prior. Mathematical Geosciences, 52(1):53-79, 2020.
12

---

Published as a conference paper at ICLR 2023
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? In ICLR, 2019. URL https:// openreview wwt wwwp///oorrdddl.leee....e. widhh1xwNhhhCmm
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Erkki Oja and Zhijian Yuan. The fastica algorithm revisited: Convergence analysis. IEEE transactions on Neural Networks, 17(6):1370-1333 1381, 2006.
Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas Information Theory, 1(1):39-56, 2020.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. arXiv preprint arXiv: 2017.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the JEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Youcef Saad and Martin H Schultz. Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal scientific and statistical computing, 7(3):866-86,, 1986.
Lisa Scarpace, I. Mikkelsen, T Cha, Sujaya Rao, Sangeeta Tekchandani, S Gutman, and D) Pierce. Radiology data from the cancer genome atlas glioblastoma multiforme (egga-gmm collection. The Cancer Imaging Archive, 1 11(4::1, 2016.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9243-9242502 9252, 2020.
Ali Siahkoohi, Gabrio Rizzuti, Mathias Louboutin, Philipp A Witte, and Felix Herrmann. Precondi- tioned training of normalizing flows for variational inference in inverse problems. arXiv preprint arXiv:2101.03709 2021.
Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening and coloring batch transform for gans. In International Conference on Learning Representations, 2018.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint /rXiv:21/1.08005 2021.
Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated mri recon- struction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 64-73. Springer, 2020.
Jean-Luc Starck, Eric Pantin, and Fionn Murtagh. Deconvolution in astronomy: review. Publica- fions of the Astronomical Society of the Pacific, 1 (000000005111 2002.
Albert Tarantola. Inversion of seismic reflection data in the acoustic approximation. Geophysics, 49 (8):1259-1266, 1984.
Jeroen Tromp, Carl Tape, and Qinya Liu. Seismic tomography, adjoint methods, time reversal and banana-doughnut kernels. Geophysical Journal International, 160(1):195-216 2005.
Dave Van Veen, Ajil Jalal, Mahdi Soltanolkotabi, Eric Price. Sriram Vishwanath, and Alexandros G Dimakis. Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv: 1806 06438, 2018.
13

---

Published as a conference paper at ICLR 2023
Jean Virieux and Stéphane Operto. An overview of full-waveform inversion in exploration geophysies. Geophysics, 74(6): WCCI-WCC26. 2009.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy, Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the Euro- pean conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with flow-based noise model. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Pro- ceedings of Machine Learning Research, pp. 11146-11157. PMLR, 18-24 Jul 2021. URL
httpss://rroceedings ml wwl..press/vlle1 http/////w//aaanlla......cc.
Tom White. Sampling generative networks. arXiv preprint arXiv:1609.04444 2016.
Jonas Wulff and Antonio Torralba. Improving inversion and generation diversity in stylegan using gaussianized latent space. arXiv preprint arXiv:2009.0653 2020.
In-Kwon Yeo and Richard A Johnson. new family of power transformations to improve normality or symmetry. Biometrika, $7(4):954-959 2000.
Özdogan Yilmaz. Seismic data analysis, volume 1. Society of exploration geophysicists Tulsa, 2001.
Jure Zhontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, et al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprint 1rXiv:181.00883 2018.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. 2018.
Rui Zhang and John Castagna. Seismic sparse-layer reflectivity inversion using basis pursuit decom- position. Geophysics, 76(6):R147-R158 2011.
Hongkai Zhao. A fast sweeping method for eikonal equations. Mathematics of computation, 74(250): 603-627, 2005.
Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions mathematical software (TOMS), 23(4):550-566 1997.
14

---

Published as a conference paper at ICLR 2023
A BACKGROUND OF FORWARD MODELS
A.1 COMPRESSIVE SENSING MRI
The MRI process essentially samples the spatial frequency components of target, following some trajectories in the spatial frequency space (k-space) according to the design of the physical system. For example, system may sample the k-space line-by-line horizontally/verticallly or in radial directions. If the k-space has been fully sampled on Cartesian grid, one can directly use inverse FFT to reconstruct the image. For various practical reasons, however, collection process, usually by skipping data points in the k-space, which can be mathematically represented by masking operation. In addition, there can be multiple coils with different sensitivity maps collecting data simultaneously. The mathematical formulation reads
necessary to speed up the data
d; PFS. te, i= Naoils
(13)
where d, E CC is the k-space data corresponding to the i-th coil, m E I. is the target object, P NxN is the mask, F e CN is the Fourier transform operator, S E CN is the point-wise sensitivity map (a diagonal matrix) corresponding to the i-th coil, and denotes noise.
To ensure fair comparison with prior work and reproducibility, we used the same masks from the repository of Kelkar & Anastasio (202T) (Fig. (7). In addition, we also used the same single-coil setup as in Kelkar & Anastasio (2021). where the sensitivity matrix an identity matrix.
(b)
Figure 7: Masks for compressive sensing MRI (Kelkar & Anastasio 2021) White: 1, black: 0. (a) 8x acceleration; (b) 2 accelcration.
We condense the effects of all operators into a linear operator A E CMN and arrive at the under- determined system
Am +
10
from the main text, where we use Acel N/M to denote the acceleration ratio.
A.2 DEBLURRING
The mathematical model behind deblurring is
+
(14)
where H is a smoothing filter, denotes convolution, and is noise.
The purpose of deburring is to recover the original sharp image m given a noisy blurred observation d.
In this study, we showed deblurring examples for natural images. In scientific applications, deblurring or deconvolution is also powerful tool. For example, in astronomy, d is a blurred image from telescope, H a point-spread function (PSF) constructed from the physies model of the telescope, and we want to obtain sharper image from the observation (Starck et al. (002). In geophysics, d can be the seismic data, is calibrated wavelet, and we want to obtain sharp images of reflectivities defining the boundaries of subsurface strata (Lines & Treitel 1984 Zhang & Castagnal 2011). In general, H is a low-past or band-pase filter, so certain frequency contents are lost in the forward process. The deblurring deconvolution process needs to recover such missing information. In addition, the noise makes the inversion process unstable. The deblurring or deconvolution problem is thus ill-posed.
15

---

Published as a conference paper at ICLR 2023
(b)
1600
Rnceiver arayy
1580
Welocity dbo (mbb)
Receivnr arra
Trovel mime mm 2ooo
1560
1540
Source
Receiver acray
1500
0.20
0.38
Twe time (mi liseordd
0.30
200
200
4:00 500
800
Receiver indlo
Figure 8: Experimental setup for cikonal tomography. (a) The target image and source locations; (b) Travel time field and the receiver array corresponding to the source. The contours wavefronts. The propagation waves can be viewed as curved rays traveling in directions perpendicular to such wavefronts; (c) The profile of the shortest
travel time recorded the receiver array. The receiver index starts from 0 in the top-left corner in subfigure (b) and increases in the clockwise direction.
A.3 EIKONAL TOMOGRAPHY
In acoustic wave imaging (e.g., ultrasound tomography), we excite waves using sparsely distributed sources one at time at the boundary of the object. Then we reconstruct its internal structures (the spatial distribution of wave speed) given the first-arrival travel time recorded the boundary. The following eikonal equation describes the shortest travel time T(x; x.,) that the acoustic wave emerging from the source location I, takes to reach location inside the target object in the figgfrreyency limit (Yilmaz) 2001)
VT (x; x.) = 1z)(x), T(x;; = O,
(15)
where c(œ) is the wave propagation speed at each location. Both this eikonal PDE and the implicitly defined forward mapping c(æ) + T'x)) are nonlinear.
To be more specific, we show the setup of our experiment in Fig. g a). For the convenience of numerical testing, put sources and receivers the boundaries of square box that contains the object, suggesting that the object is immersed in square box filled with water. In reality, one can put sources and receivers directly on the target. The dimension the square area 25.6 em x 25.6 em with grid interval of 0.001 m. There are eight sources located on each side, and receivers are located each grid point on the boundary. Fig. (b)) shows shortest travel time field the generated wave from the indicated source location. For each source, only use receivers on the three other sides, indicated by the red lines, excluding the one on which the source is located. Fig. g c) shows the profile of the shortest from 0 in the top-left corner and increases the clockwise direction.
travel time recorded at the receiver array. The receiver index starts
We interpret the nonlinearity and ill-posedness of eikonal tomography from another perspective. As shown Fig. B(), we plot contours T(x) const representing the wavefronts. Under the high-frequency approximation, the propagation of waves can be viewed curved rays traveling in
16

---

Published as a conference paper at ICLR 2023
directions perpendicular such wavefronts. Since the wavefronts depend on velocity field c(æ), the rays are also functionals of the parameter c(x) to be estimated, contrary to straight-ray tomography such CT. The eikonal inversion problem is thus nonlinear. Besides, the rays carry information about the medium averaged along their paths. One property of curved-ray tomography is that the ray coverage is uneven inside the object. In fact, curved rays tend avoid low-velocity areas, giving us little information about such regions, making the inverse problem intrinsically ill ill-posed.
We solved the eikonal equation using the fast sweeping method (Zhao 2005) and computed the gradient using the discrete adjoint-state method, using the code from the repository oi et al. (2020).
Our eikonoal tomography uses the same StyleGAN2 network as the compressive sensing MRI experiments. The value range of StyleGAN2 is [-1, 1]. In the forward model, we map the StyleGAN2 output c(æ) in steps. First, we convert its values the range of [0, 1] by using m (m++1//2
Second, we convert the values to the range of acoustic wave velocity using m +- 100 x m + 1500. This relationship is purely manufactured for our synthetic tests. One should use more realistic relationship in practice.
B ADDITIONAL MOTIVATING EXAMPLES
Glow We varied the fg-norm of the latent tensors for Glow and reported the outputs in Fig. 9 The images are getting increasingly unrealistic as we increase the norm from 1.0vd, where d is the latent tensor dimension. This phenomenon demonstrates why we need to use a temperature < for Glow.
0.1vd
0.7vd
100d
2.0vd
Gaussian
Skewed
Figure 9: The visual effects of the l2 norm of latent tensors Glow outputs. The top and bottom panels correspond to latent tensors drawn from a spherical Gaussian and a spherical skewed distribu- tion, respectively. The two tensors are scaled to have various norms: 0.1, 0.7, 1.0, and 2.0 of v (the square root of the latent tensor dimension). The images getting increasingly unrealistic increase the norm from 1.0vd. This phenomenon demonstrates why need use temperature < for Glow.
StyleGAN2 StyleGAN2 is well-designed generator with built-in spherical transformation for the style vector. seems very robust to random vectors drawn from the distributions similar to those in Fig. [ However, were able to find challenging examples for it from inversion tasks. In Fig. [0 The first example is the direct output only using the pathologic style vector. The second example is the generator output using the style vector after whitening only. As ineffective improving the image quality, which is confirmed by our ablation studies. On the contrary, if we the ICA layer, see huge improvement in visual quality, and interpretation is that getting rid of higher-order dependencies is very important. Note that there are eyeglasses in the third figure, meaning that if we relax the 1D Gaussian requirement, we can sample some rare examples. Finally, we use the full G layers to get another plausible image.
see, whitening alone
Stable diffusion Stable Diffusion (Rombach et al. 2022) is state-of feffthe-art art deep generative model with text-to-image synthesis capability, which maps a Gaussian latent tensor to a high-resolution image.
17

---

Published as a conference paper at ICLR 2023
processing
Whitening only
ICA only
Full layers
Figure 10: The visual effects of various components of Gaussianization layers on pathologic style vector for StyleGAN2.
Before
After
Before
After
Before
After
Skewed
Non-i.i.d. Heavy-take tailed
Figure 11: Comparisons of Stable Diffusion outputs before and after applying the Gaussianization layers to latent tensors. The components of these latent tensors are i.id. skewed, i.i.d heavy- tailed, and non-i.id., respectively. We scaled all latent tensors be the sphere with radius of Vtensor dimension. Stable Diffusion picks up and amplifies the sparse large-an amplitude points in the heavy-tailed case and the 2D sinusoidal pattern in the non-i.i.d. case.
0.7Vd
Vd
1.3vd
0.7
Vd
1.3vd
Figure 12: Comparison of Stable Diffusion outputs generated using spherical Gaussian latent tensors of different norms, where d Vtensor dimension.
18

---

Published as a conference paper at ICLR 2023
We repeated the tests for Glow and StyleGAN2 Stable Diffusion and reported the results in Fig. [1 The distribution for skewed case was the exponential-ganee distribution log [(1, 1) (or the "log-gan gog-gamma distribution" in scipy), the distribution for the heavy-tailed case was Lambert W x Fx distribution with parameter 5 = 0.5 based on a standard Gaussian (Goerg, 2015). To generate the latent tensor with non-i.i.d. entries, we first sampled 2n R N NO I). Then we added 0.5 * sin (2TT 22///4 x) & cos(2n) / /64 y) to each channel of Zo for the latent tensor z, where @ stands for outer product. The number of denoising steps, classifier-free guidance, and the output dimensions for Stable Diffusion are 50, 7.5, and 512 x 512, respectively.
The outputs of Stable Diffusion are completely out-of-range the latent tensors deviate from standard Gaussian, with much poorer quality than those from Glow and StyleGAN2. For example, Stable Diffusion picks up and amplifies the sparse large-amplitude points in the heavy-tailed case and the 2D sinusoidal pattern in the non-i.i.d. case. However, with our Gaussianization layers, we are still able to constrain the outputs within the plausible range. Interestingly, we show in Fig. [2]tta Stable Diffusion is much more sensitive to the norm of latent tensors than the other DGMs. Therefore, we anticipate that our Gaussianization layers are critical for potential inversion applications using Stable Diffusion such text-guided inversion.
C ADDITIONAL EXPERIMENTS
First, we report in Table an ablation study on the effects of various combinations of regularization techniques on style vectors and noise maps in StyleGAN2. The observations are (1) If we turn off the update of noise maps, the Gaussianization layers lead to the best result; (2) If we turn on the update of noise maps, applying the Gaussianization layers to both the style vectors and noise maps gives the best results.
Table 5: Ablation study on different combinations of regularization on the style vectors and noise maps in StyleGAN2 for compressive sensing MRL In the parentheses, "u" stands for unconstrained, "s" stands for spherical constraint, "g" stands for reparameterization with the Gaussianization layers, "o" stands for orthogonal reparameterization, and "X" means that the update is turned off. Note that StyleGAN2 has a built-in spherical transformation layer for the style vectors.
Acel = 8x, SNR = 20 dB
Acel 8x, SNR = 10 dB
Method
PSNR
SSIM
PSNR
SSIM
Style (8), Noise (X)
26.5843300
0.80
25.57
0.78
Style (g), Noise (X)
27.39
0.83
27.02:1457
0.8210.12
Style (o). Noise (X)
25.88+534
0.78
25.39
0.77+
Style (s), Noise (u)
26.5 1 1:6.1
1.776
22.49
0.57
Style (g), Noise (o)
27.43
0.81
24.93
0.76#00122
Style (o), Noise (o)
26. 145779
0.78
20.104500
0.77 132
Style (o), Noise (g)
26.04
0.78
25.22
0.77 4-0.13
Style (g). Noise (g)
27.99
0.83
25.48
0.78
Second, we studied the effect of patch size on the performance of Gaussianization layers. To be more specific, we tested the effects of various patch sizes on 1D style vectors. The largest patch size is 64 since the number of extracted patches should not be smaller than the dimension of the patches. We only turned on the ICA layer and the standardization layer in this experiment. One can observe that both the PSNR and the SSIM increase as the patch size increases. We would advise using the largest possible patch size in Gaussianization layers, although we only used patch size of 32 for style vectors in all other experiments.
In all experiments, we fixed the patch size for Glow 3 x 8 x 8 and the patch size for the noise maps in StyleGAN2 as x 8 x 8, In the experiments with Glow, the image dimension was 3 x 128 x 128. If we chose larger patch size, we could not obtain enough patch vectors. As for StyleGAN2, if look at the parameterization illustrated in Fig. [4, there is already a 4 x noise map unconstrained choose the patch size as 8 x 8. If we increase the patch size to 1 x 16 x 16, there will be two
19

---

Published as a conference paper at ICLR 2023
additional 8 8 noise maps unconstrained. To minimize the influence of unconstrained parameters, we chose the patch size for noise maps as x x
Table 6: Ablation study on patch size of the style vectors in StyleGAN2.
Method
PSNRT
SSIMT
Patch size = 8
25,33:1.12
0.76
Patch size 16
26,83:1657
0.79
Patch size = 32
27.91
0.82
Patch size 64
28.03
0.83
Third, we did parameter sweep on the weighting parameter 3 in the Glow-regularized deblurring problem (formulation 3 or [4).. Consistent with Fig. 20 the conventional Glow-based regularization is not as effective as our Gaussianization layers. If B is too large (e.g., 10 or 100), the inversion results become unrealistic with huge errors.
Table 7: Glow-regularized deblurring results using different [s..
Method
LPIPS
PSNRT
SSIM
0.0
0.172
21.74
0.58
B 0.01
21.75 0.172
0.58
3 0.1
0.17420.66
21.7221.15
0.57
9 1.0
0.1742n06
21.95
0.59
3 = 10.0
0.247
22.55
0.58
3 = 100.0
0.602
12.56+263
0.12
Fourth, investigated the performance of Gaussianization layers when the forward model is inaccurate (Table 8). Our Gaussianization layers still outperform the conventional Glow-regularized inversion.
Table reow-regul deblurring with an inaccurate filter. The ground-trus truth standard deviation of the Gaussian filter is 3, which is used to generate the observed data, but used 5 for inversion.
Method
LPIPSJ.
PSNRT
SSIM1
Conventional (3 1.0) 0.21 4006
17.72 0.49
G layers
0. 0. 18.70
0.50=2.000
In addition, we did an ablation study on the components of Gaussianization layers in Glow-regularized deblurring (Table Pp. We adopted the second parameterization scheme (App. D). The observations are similar to those from Table 2 1. The ICA layer is the most significant part in improving result
scores, especially LPIPS the score that matches human perception the best; The whitening/ZCA layer is not effective when used alone; 3. There is clear winner among combinations of ID Gaussianization layers, and their difference in performance is marginal. So we picked the Lambert W x Fx layer, one of the best-performing options, in all other deblurring experiments the ID Gaussianization layer.
Inversion using out-of-distribution images Finally, we tested the limit of DGM OGM-regulrrized inversion by using real-world out-of-distribution target images. We randomly sampled 25 MRI images from the fastMRI DICOM dataset (Zbontar et al.) 2018 Knoll et al., 2020). The RSNA clinical trial processor was used to anonymize the whole dataset, and each image has been manually inspected to make sure that there is no protected information. According to the fastMRI paper (Zbontar et al.] 2018), "This dataset represents larger variety of machines and settings than present in the raw data." Also, the image values are represented uint rather than float numbers, and their value range can be very large (0 to several thousand). So normalized the image values be
20

---

Published as a conference paper at ICLR 2023
Table 9: Ablation study of the components of Gaussianization layers applied to Glow-regularized deblurring.
Method
LPIPS
PSNRT
SSIM1
ICA (X).
YJ (x).
Lambt (X)
0.17
21.79+1111
0,583++0066
ICA (X),
(X),
Lambt (v)
0.16
21.96+131
0.588
ICA (X),
YJ (v), Lambt (X)
0.17
21.75
0.579
ICA (X),
(V). Lambt (V)
0.16+0as
21.994+120
0.587+0006
7CA,
YJ (X),)
Lambt (x)
0.16
21.666+:228
0.579
ICA (V).
YJ
Lambt (x)
0.1 0.14
22.5
0.586
ICA (V), (X),
Lambt (v)
0.13 13
22.47:2122
0.590
ICA (V). YJ (V).
Lambt (x)
0.1 t.14 22.49
0,586
ICA (V). YJ
Lambt (V)
4.13 22.40+127
0.586=+0.06
() / img.min 1. However, the trained StyleGAN2 outputs are always slightly beyond [-1, 4], and we clipped the values in both test image generation and inversion. Therefore, the new examples are out-of- distribution.
between range [-1,1] using img (img
(img max () -img min ()
We kept the same experimental setup 8x acceleration and 20 dB SNR and tested inversion using the spherical constraint and Gaussianization layers on both only using the style vectors and using the additional noise maps. We report the inversion results Table and Fig. [3] To save experiment time, we only use one ID Gaussianization layer (either Yeo-Johnson or Lambert W x Fx).
Table 10: MRI inversion using real-world out- dfitt---iittitttion images.
Method
PSNR
SSTM
(a) Style only, spherical
26. 3.5
0.784
(b) Stylc+noise, spherical
25.26
0.755
(c) Style only, G layers, YJ for ID
0.765 2..33
(d) Style noise, G layers, YJ for ID
25.39
0.769 200045
(e) Style only, G layers, Lambert for ID
25.39
0.762
(f) Style + noise, G layers, Lambert for 1D
25.45
0.765
First, none of the inversion results shown here are visually comparable to the ground truth, which shows the necessity of having a large training dataset that covers the target distribution. Also, it is vital to ensure that the pre-processing and value ranges of training and target images are the same. Second, in terms of metrics, the results are comparable to but slightly worse than those on synthetic test images reported previously. This is expected because these images are out of distribution. Besides, in cases where we applied Gaussianization layers, using both noise maps and style vectors gives better results than using style vectors only. In addition, Gaussianization layers improved results when also optimized the noise maps in addition to the style vectors. However, we found that only using the style vector and the spherical constraint gives the best result this test set, which contradicts results reported in Ta Our interpretation is that the dimension of style vectors is still relatively lower than noise maps or the latent tensor used in Glow, our Gaussianization layers may be more effective in dealing with the latter two types of parameters.
D REPARAMETTRIZATTION SCHEMES FOR STYLEGAN2 AND GLOW
Fig. 14 illustrates the reparameterization scheme for StylgeGAN2 using the Gaussianization layers. The patch size for the style vectors and noise maps are 32 and x 8 x 8, respectively. Note that we transpose the latent tensors in Fig. 14 compared to the notations in corresponding equations and algorithms. Also, pointed out in Gu et al. (2020), also find that single latent code is insufficient for image reconstruction. Thus, we use multiple style vectors that were fed into different intermediate layers.
21

---

Published as a conference paper at ICLR 2023
True
(a)
(b)
(c)
(d)
Figure 13: Examples of MRI inversion using out-of-distribution images. (Accl=8x, SNR=20 dB). (a) Style vector only spherical constraint; (b) style vector noise maps spherical constraint; (c) style vector only + Gaussianization (only the Yeo-Johnson layer for ID Gaussianization): (d) style vector + noise maps + Gaussianization (only the Teo-Johnson layer for ID); (e) style vector only + Gaussianization (only the Lambert layer for ID); (f) style vector + noise maps Gaussianization (only the Lambert layer for ID).
Noise Parameters
Style Parameters
2730
Reshape
14 x 512
Noise Parameter x
Gaussianization
Style Vectors
Noise 8 x
Gaussianization
224 x
Noise x
2 x 64 8 x 64
StyleGAN2 Generator
Unfold
Noise 16 x
Style Vectors
Noise x
20-18
Reshape
2730 64
Style Vectors
Noise 256 256
14 512
Noise 256 x 256
Image
x 256 x 256
Figure 14: Reparameterization scheme for StyleGAN2. The dimensons of the latent parameters are [the number of vectors vector dimension), except the one. Note that we transpose latent tensors in corresponding equations and algorithms. The style and noise parameters before the Gaussianization layers are {vili=t,... ,N) and the style and noise vectors after the Gaussianization layers are (zliii,....N mentioned in the main text.
22

---

Published as a conference paper at ICLR 2023
(a)
(b)
Parameters
Parameters
192 x 256
192 256
Ganssianization
Gaussiaizzation
Patch vectors
Patch vectors
192 x 128, 32, 32)
192 x 256
Unfolding
Unfolding
Latent tensor
3 x 128 x 128
Latent tensor list
Multi-scale Squeezing
x 64 64
12 x 32 32
Latent tensor list
24
16
6 64 64
96 8 x 8
12 32 x 32
24 16 x 16
Glow
96 x 8 x 8
Image
Glow
128 x 128
Image
3 x 128 x 128
Two reparameterizztion schemes for Glow. (a) Pl; (b) P2. The dimensions of the latent parameters are [vector dimension x the number of vectors]. The parameters before the Gaussianization layers are (velk=1,.....> and the patch vectors after the Gaussianization layers are (zi,i=,, v mentioned in the main text.
Figure 15:
23

---

Published as a conference paper at ICLR 2023
Tensor (1 channel)
Tensor (4 channels)
1
2
3
4
6
8
5
7
8
Squeezing
14
16
10
11
12
13
15
2
4
13
14
15
16
10
12
Vectors
Tensor
V1
Unfolding
Vi
V2
V3
V3
V4
Figure 16: Illustration of the squeezeing and unfolding operations.
For Glow, we came up with two reparameterization schemes (Fig. 15). The first one (PI) reparame- terizes patches from latent tensor with the same dimension as the output image. The dimensions of the patches and the image are 8 x 8 and 3 128 128, respectively. Then a multi-scale squeezing operation maps the latent tensor into a list of tensors corresponding to different scales of Glow. Glow uses the list of tensors as the input. The second scheme (P2) reparameterizes patches extracted directly from tensors in the above list. We also partition each tensor into 3 x x 8 patches. The unfolding and squeezing operations are illustrated in Fig. T6] the multi-scale architecture of Glow, the squeezing operation is applied recursively on half of the output tensors cut in the channel direction (Dinh et 2016 Kingma Dhariwal 2018).
E
MORE DETAILS OF THE GAUSSIANIZATION LAYERS
E
ICA LAYER
The overall ICA layer is summarized algorithm [ We maximum number of the fixed-por point iterations to reduce computational cost and ensure accurate gradient computation that can pass the finite-difference convergence test. The gradients are backpropagated through the loops without using the trick introduced in App.
Whitening The FastICA algorithm typically requires that the data are pre-whitened. Let V be the data matrix whose columns are data vectors. We first subtract the mean from the data vector using V t-
mean(V, dim = =). Then we compute the data covariance matrix using C = (1 n) N-I VVT + III where we use small constant (e.g.. 0.001) to blend the empirical covariance matrix and an identity matrix to avoid ill-conditioning.
After these data preparation steps, the ZCA whitening used throughout the study first computes the eigenvalues and eigenvectors D. and then output the whitened data using V DA-1/22 D V.
24

---

Published as a conference paper at ICLR 2023
Alternatively, the following steps to whiten the data, which are also used later in ICA iterations to decorrelate column vectors in the orthogonal matrix (Hyvarinen 1999):
1. Initialize W
2. Compute
W I/T
(16)
3. Repeat until convergence
W + W 2 CW,
(17)
4. Output V < W
Algorithm 1: ICA Layer
Input Data matrix V E RXN, error tolerance e; damping parameters: 10 and
== 0.8; maximum iteration numbers J 10 and K 100.
Output Matrixx P E RxN with i.i.d. entries
// Whitening stage
V = mean = 1), C= (1- n) n) +nI:
V < ZCA PCC--wiitening//
// ICA stage
W = W I, j j+1;
while j S J do
W + N aVo W v) Wdiag (o' (WT T V) 1) Eq.
W = Wo + W/
W W k 1;
while K do
W 2 W
WW W
if W Woll < then
10
break;
11
Wo W, k+k+1;
if W WI|l < then
13
break;
14
W W, j -+1+1;
return P W VV.
15
The modified FastICA iterations. As stated in Hyvärinen (1999). neural unit the weight vector w; and input v is
the objective function for one
arg max E T (w, v)] s.t., E (w, v) [ =1, W0;
(18)
where the contrast function (e.g., logcosh). The original derivations convert this constrained optimization to an unconstrained one using Lagrange multipliers. However, this procedure is unnecessary since the matrix W is orthogonalized after each iteration, and the input vectors have been writeeiee Therefore, we only need to solve the following equation
E vo (w. [Tv] 0,
(19)
whose Jacobian is
J = Elv El (w, v)
(20)
E vv [[o(w, v)] = (ww v)].
where e is the derivative of Φ The Newton iteration scheme is thus
un; wi E[vo (.:) /R v)].
(21)
25

---

Published as a conference paper at ICLR 2023
To improve the convergence, damp the iterations by a parameter (0. Also, using the same technique to convert the Newton iterations to fixed- -ointt iterations in Hyvärinen (1999); Hyvarinen (1999). arrive at the modified fixed-point iteration scheme:
wd
aE (v ( v)] [Ø (w, v)] wi
(22)
followed by the aforementioned decorrelation procedure after each step. The convergence of the modified FastICA iterations can be proved similarly in Oja & Yuan (206)).
E.2 POWER TRANSFORMATION LAYER
Algorithm 2:
Power Transformation Layer
Input Data vector p
Output Vector whose values are 1D Gaussianized.
Estimate À from p using Eq. 24]
Compute s with the estimated and data p using Eq.
return
We propose use the power transformation or Yeo-Johnson transformation to reduce the skewness of distributions:
Yeo & Johnson 2000)
(p + 1)* -1) /x,
020,0
log (p + 1);
P20,A=00
s\.
(23)
11-p+ 112 - 1)/ (22- A)],
,
log(-p++ 1++1) 1),
P<0,A= =2,
where p an input value, s an output value, and the parameter to be estimated. As shown in Fig. Ba), the form of the Yeo-Johnson activation function depends on parameter X. If X = 1, the mapping an identity mapping. If 1, the activation function is convex, compressing the left tail and extending the right tail, reducing the left-skewness. If X < 1, the activation function is concave, which oppositely reduces the right-skewness. The only parameter determined by solving an optimization problem that minimizes the negentropy:
X = arg max / (Alp) = arg max log Var (s (A,) + (A (A- Σ sign log(Pdd + 1),
1
A
(24)
where p is the input data vector with entries Pi, i=.
We custom operator based SciPy's implementation using Brent's algorithm (Brent 2013) to find an approximate minimum of Problem 24 Continuing from the approximate minimum, we use Brent's root finding algorithm (Brent 2013 to find the minimum where the gradient vanishes.
Since the parameter depends on input data, we need to back-propagate the gradient through the optimization process.
The power transformation layer is summarized in algorithm
E.3 LAMBERT WA Fx LAYER
Due to noise and inaccurate forward models, we observe that the distribution latent vector values tends to be shaped heavy-tailed distribution during the inversion process. To reduce the heavy- tailedness, we adopt the Lambert W x Fx method detailed in Goerg (015).
Let X be random variable whose CDF is Fx, with mean μAx and standard deviation ox. The following transformation with a heavy-tail parameter 2
{ ) 2
S = (Uexp
+ px:
(25)
26

---

Published as a conference paper at ICLR 2023
where U = (X px) /ox, is bijection and maps X to another random variable S with heavier tails.
The transformation Eq. 25 is bijective if ò 2 0, and we
the Lambert W function to find its
inverse. The Lambert W function W is defined as the inverse of q = W-1 (t) exp (t), where t and g scalars. Given q. Halley's method can be used to find t W(g) (Corless al. 1996).
Hence, the inverse of Eq 25 is
( (S-Px px
X= Ws
ox + px,
(26)
OX
where
W (du²)
Ws (u) sign (u)
(27)
We use the parameterized Lambert W x Fx distribution family to approximate heavy-tailed input distribution and use Eq. 26 to recover a distribution with lighter tails. order to make the recovered compute the optimal heavy-tail parameter 5 by
distribution close to Gaussian distribution, minimizing the difference between the kurtosis of the output distribution and (Kurtosis a common surrogate measure of negentropy (Hyvärinen & Oja. 2000)):
8CMM = arg min Kurt W.
ux
))
2
(28)
8>0
Ox
where is the data vector, and Kurt is the kurtosis. We constrain 5 > O, and solve Eq. 28 using the L-BFGS-B optimizer (Zhu et al. 1997).
In addition, estimate the mean μ.x and standard deviation ox along with ó using the Iterative Generalized Method of Moments (IGMM) (Goerg 2015), which embeds an optimization problem for 8 in outer loop of iterations to estimate o and μx (see algorithm 3). If the kurtosis of input data vector is not greater than 3, we skip the whole Lambert W x Fx layer by directly outputting the data vector.
Algorithm 3: Lambert W x Fx Layer with the Iterative Generalized Method of Moments (IGMM) (Goerg 2015)
Input Data vector s, error tolerance c 10-5, maximum iteration number 100. Output: vector æ whose empirical distribution is less heavy-tailed than and its kurtosis Rs 3)
Initialize: g(0) + [xX) (0) _(0) k = 0:
2 Compute initial kurtosis 82 = Kurt(s);
if [32 S then return s; end
while
K and [|5(k) - ((k-11]| 2 do
(k) (s (1 (k)
/0 (k).
Compute ((++1) using Eq. 28 u
(k+1) Waan (ulk)): ((++1) + u (1+ (K) (K)
4
Update xx
(R+1) E [k+
1)] and ox
(k+1) / Var (I(k+1));
((++1) fx
(K+1) a (k+1) g(++1))
10
11
ls ls 11
end
13
return Φ Ws
8- ox
σx + px.
F DETAILS OF DATASETS AND TRAINING
For MRI and eikonal tomography, we generate synthetic brain images as inversion targets using the pre-trained StyleGAN2 weights from Kelkar & Anastasio (202)) which are trained on data from the
27

---

Published as a conference paper at ICLR 2023
databases of fastMRI (Zbontar et al., 2018 Knoll et al. 2020), TCIA-GBM (Scarpace et al. 2016).
and OASIS-3 [aMontagne et 2019). As a result, there is no sensitive personal information in target images. In the data generation process, only used one style vector of dimension of 512. We picked 100 images that are visually plausible, whose random seeds can be found in our code. These random seeds were never used to initialize inversion. In inversion experiments,
used 14 such style parameters: one for the lowest resolution of 4 x 4, one for the tRGB layer, and two for each resolution from 8 to 256 x 256, to avoid inversion crime. This choice of expanding latent space dimension is also justified by our observation that only one style parameter vector is generally insufficient for inversion tasks with data from datasets such as CelebA-HQ (Karras et al., 2018).
We used the CelebA-HO HQ dataset (Karras et al., 2018) (under the Creative Commons CC BY-NC 4.0 license) for the deblurring experiments. All images were downsampled to the resolution of 128 x 128. We split the 30000 images from CelebA-HQ into the subsets of training (24183 images). validation (2993 images), and testing (2824 images) following the original splits from CelebA (Liu et al. 2015). For the inversion tests, we randomly selected 100 images from the test set.
2.75
Train split: levels
2.50
Validation split: 4 levels
Train split: levels
2.25
Validation split: 5 levels
2.00
Bits
1.50
1.25
1.00
0
200
400
600
800
Iterations
Figure 17: The negative log-likelihood or NLL (reported in bits per dimension) on the training and validation splits of the CelebA-HQ dataset with different numbers of multi-scale levels.
For the pppee-parameter of the Glow networks, we used 4 multi-scale levels and 32 ftow-steps, and we only used additive coupling layers. Fig. 17 reports the training process. For each epoch, we computed the training negative log-likelihood (NLL) averaged throughout the epoch, and the validation NI.L. at the end of the epoch. The validation eurves suggest that it is better to use multi-scale levels. We chose the network weights from the epoch before the validation NLL stopped to decrease: 850 for the CelebA-Ho dataset. All training was conducted using x 32 GB Nvidia V 1000 GPUs with a batch size of 64. We used the Adam optimizer (Kingma & Ba 2015 with
learning rate of 10-4, as well as 31 = 0.9 and $3 = 0.99.
G GRADIENT COMPUTATION OF THE OPTIMIZATION-BASED DIFFERENTIABLE LAYERS
arg ming 1 (he (x))
Input: X
hgg (x)
Output: y
Figure 18: Illustration of the forward computation of optimization- based ICA, Yeo-Johnson, and Lambert W x Fx layers. We denote the input and output by x and respectively. The layer is represented by hgg and the layer defined by solving an optimization problem whose objective function is 1. The back-propngation of gradients in components defined by Eq. 24 and Eq. 28 is enabled by the implicit function theorem and automatic differentiation detailed in App. G
As shown in Fig. 18 in the power transformation and Lambert W x Fx layers, there are operators whose outputs are obtained by solving optimization problems formally described as
0 arg min (xx 0),
(29)
28

---

Published as a conference paper at ICLR 2023
10a
Asproximetion error
10-1
10
10
Finite difference
with custom gradiant
10a
1st order
2nd order
10
10"1
10-
10
Figure 19:
Gradient accuracy test for custom operators. An accurate gradient should make the error converge at rate of the second order following the red dashed line.
where denotes the objective function that defines the operator (combining the objective function and layer he in Fig. 18), symbol 0 stands for the optimal output, scalar in our cases but can also be vector general situations. The optimal condition
(0, 0) == Lax, 0) 0,
(30)
where the subscript denotes partial derivatives.
The optimal condition implicitly defines forward operator of the following form:
0 Ofoowwrrd
(x)
(31)
The backward operator is
0x o bbackward
0x
0"
(32)
Ox
00
where X is the objective function of an inverse problem. Differentiating Eq. 30 with respect æ, we have
La + Lg02
O2 -Lp' Lax
(33)
using the implicit function theorem. Then, to back prok-prop the gradient from
0x 30
to
0x Bm
we use
0x 0x 0₂
0x Lø L.
0x
H LL:
(34)
dx d0
0e
00
where Hg is the Hessian matrix x with respect to
In our problems, the output 0 is scalar, so it is easy to use automatic differentiation to compute Lo directly and hence 4 Otherwise, if has many parameters, we can first solve following linear system with auxiliary vector A:
0x 00
AHg
(35)
and then compute the gradient using
0x XLz
(36)
Fr
a technique also known as the adjoint-state method. Note that there no need to compute the Hessian explicitly, but one can automatic differentiation compute the vector-Hessian product AH, and utilize iterative linear solvers like GMRES (Saad & Schultz, 1986) to solve the linear system.
As final note, check the accuracy of our gradient computation using the fiitt-ddifference convergence test based on Taylor expansion:
f(x + cōz) f(x) + cV f(xx) dx +
(37)
where dæ is a random vector with a unit Eg norm, and ff(x) is computed using our custom gradient. We here define f as composite function that maps the (vector) output of an forward operator to a scalar, e.g., f(x) = Porooward (x: 113. Once we decrease c, we should see that the error term f(x + cōx) - f(x) - <V (x)) decreases at a rate of the second order. All our layers passed this test, the example shown in Fig. 19 This test should be conducted in double precision.
29

---

Published as a conference paper at ICLR 2023
H ORTHOGONAL REPARAMETERIZATTON
We also propose reparameterize the latent vector using orthogonal matrix R:
z Rv, v (),
(38)
where v is fixed during an inversion, and we treat R as the parameter instead. There are various ways parameterized orthogonal matrix. We choose the Cayley parameterization:
R = (I W)(I - W) where W = =
(39)
one of the best reported in Liu al. 2021). Therefore, the DGM-regulantn inversion using orthogonal reparameterization is
arg min (1/2) [|d g (Rv)
(40)
The specific reparameterization schemes for StyleGAN2 and Glow are the same as in App. For style vectors, the full dimension of 512 because of the observation described in Table For
Glow and noise maps in StyleGAN2, we use patch sizes 3 x x 8 and 1 x x 8, respectively, to save computation time and memory.
I GAUSSIAN TYPICAL SET
The Gaussian annulus theorem states that a high-dimensional standard Gaussian distribution has most of its probability mass concentrated within an annulus area around high-dimensional sphere:
Theorem (Gaussian Annulus Theorem (lumm et al. 2020 For an n-dimensional standard
Gaussian, for any B < vn, all but at most 3e -5 of the probability mass lies within the annulus + B, where is a fixed positive constant.
vn- 3
This theorem gives necessary geometric condition of typical samples from high-dimensional standard Gaussian. However, the converse is not true: not all vectors whose & norm is / B S |x| S vn+ are typical examples from the standard Gaussian distribution. As result, a DGM does not necessarily map latent vector staying within the Gaussian annulus geometrically to plausible image, which demonstrated in Fig.
The formal definition of a typical set is as follows.
Definition (Cover & Thomas (012)) Let px(x) be distribution whose support is X. The typical set A defined as the set of sequences "2,""" E X Wi ~ px, that satisfy
1
[
logpx (#1, ... , Se.
(41)
71
where H [X] is the entropy random variable X.
Now random vector R (0, o²I) can be factorized i.i.d. random variables that distributed as (N o²). Therefore, we can regard x as an i.i.d, sequence and give the following definition:
Definition (Gaussian Typical Set) Gaussian typical set is the rypical set A R" R N(0,o21)
The following theorem guarantees that a typical sample from x E T ~ N(0,021 resides in the Gaussian typical set with very high probability.
Theorem (Coerr& & Thomas (2012)) For every e > O, the typical set has probability P (A!))
1 with sufficiently large dimension n.
30

---

Published as a conference paper at ICLR 2023
This theorem is direct application of the asymptotic equipartition property (AEP), which is based on the i..d. assumption of sequence entries and the weak law of large numbers. Similar to the Gaussian annulus theorem, Theorem 2 depicts geometric property of the Gaussian typical set: it is concentrated in an annulus near a shell of radius a y,, which can be directly verified by definition. Of course, equivalently, if vector whose & norm significantly deviates from v cannot be typical sample from a standard Gaussian. However, one cannot assert that a vector is sampled from the Gaussian typical by only checking its norm.
J MISCELLANEOUS TOPICS
Original
Observed
0.01
0.1
8 1.0
B 10.0
Ours
UN
Figure 20: Comparison of conventional DGM (Glow)-regularizz) deblurring results with different weighting factor Bs and the result using our Gaussianization layers.
J.1 INVERSION USING GLOW
In the Glow-regularization framework, we solve the inverse problem by finding the maximum a posteriori (MAP) estimate from
PM (m|d) x (d|m) (m),
(42)
where M denotes the parameter space. The probability density par introduces priori knowledge and is represented by a normalizing flow (Glow) ge. which a differentiable invertible mapping between two distributions, parameterized by neural network parameters O: m go where is the latent vector. After training, the log probability density of a given model m is
log pas (m; 0) log pz (90" (m)) + log det J (m) log pz (z) log det Jnp (z)), (43)
where Z stands for the latent space. When we use the trained network in inversion, we freeze the network weights; hence, we drop go hereafter in our notation. Therefore, Eq. for Glow- regularized inversion is
min (1/2) ||d fog (2)1 -3 (log Pz (z) log det Jy (z))))
(44)
x
The new regularization term in Eq. 3 is R'(z) = - (log Pz (z) - log del Jz(z))) and we have shown the results with different As in Fig. 20 and Table 7
J.2 DUALITY OF KL DIVERGENCE
As also shown in Papamakarios et al. (211)) the KL-divergence between two distributions does not change under a differentiable invertible transformation, so
DEL [py (m) a (m; 0)] DRL Ip ())p)))))
(45)
where PM is the target distribution in the physical parameter space, and pz is the corresponding latent-space distribution under the normalizing flow. This means that minimizing the forward KL divergence in the M domain or physical parameter space is equivalent to minimizing the reverse KL-divergence in the Z domain the latent space.
This fact and Theorem 2 imply that a well-trained normalizing flow maps samples from the target distribution into the Gaussian typical with very high probability and vice versa.
31

---

Published as a conference paper at ICLR 2023
J.3 INVARIANCE OF KL-DIVERGENCE AND MULTI-INFORMATION
For the completeness of this paper, we briefly prove the key properties of KL-divergence and multi-information used in the Gaussianization framework.
First, we prove that the Kl--divereence invariant under differentiable bijections. We write the definition of the KL-divergence between distribution Px and 9x of random vector
DRI((xxllxe) f
Px(x) da.
PP px((x) log Ix (x(x)
(46)
Suppose there is a differentiable and invertible transformation T that maps to u. u == T(x). Then, we know the PDF under change of variable is
(x Pu (u)) det Jr (x)),,
4x (x(x)) qu(n)) det Jr())))
(47)
where we define the Jacobian matrix Jr(x))
du
(z). Therefore, have
8x
(PI(lxllox)) = / px((0) log
(x da (x (x)
(48)
/ Pu(u)) det (x) [ log
Pu(u)) det Jr (x)) Yu (u(u) det Jr))) de
DKL (Pullgu)
Then, we show that the multi-information does not change under component-win wisse invertible differ- entiable transformations. The multi-information is defined by the KL-divergence between the joint distribution p(x) and the marginals:
D
[(x) = [xx1 ... ......",, ,x) = DKL
(xx)
[x,)
(49)
We define the bijection Tx) specifically with invertible differentiable transformations each component (xx) u,- Then, using the Ki-divereence invariance just proved and that
IЛ (x,) = II pá (u,) (z)) we get ((x) I(u).
J.4 THE ROLLING OPERATION
After applying one set of the Gaussianization layers to the latent tensor, might be desirable apply the layers to different set of non-overlapping patches from the latent tensor again. The rolling operation shifts the latent tensor in the horizontal and vertical directions by half the patch size w before patch extractions. The values at the boundaries wrapped around to the opposite sides. One can, for example, use the torch. roll command to implement this functionality.
K ADDITIONAL DISCUSSIONS
Original
Observed
G layers
Figure 21: A failure example where inversion is unable to restore the eyeglasses because of the strong constraint from traversing within the Gaussian typical set. The results are dominated by the bias of the training dataset.
We point out the limitations and potential improvements of this work. First, requiring latent vectors to traverse within the Gaussian distribution/typical set means that the statistics of the training dataset will dominate the results. Fig. 21 shows that our method cannot restore the eyeglasses because
32

---

Published as a conference paper at ICLR 2023
it is not a significant feature within our training data. One needs to pay particular attention to training dataset construction to ensure that it represents the typical features to be investigated. One should also interpret their results with this caveat in mind. On the other hand, one may combine Gaussianization layers and latent space manipulation techniques (Shen et al.] 2020) for a guided inversion, e-g., requiring that the inverted images should have eyeglasses. Second, we only use one set of Gaussianization layers, One can further improve the performance by using additional sets of layers, with possible different partition schemes, such using the rolling operation (App. [.4] to shift the patch extraction locations. Third, since the deep generative models are highly nonlinear, the results may get stuck in local minima. We mitigate this problem by starting inversion from multiple randomly initialized latent vectors. However, this technique is computationally expensive. It is an important research direction to improve the initialization process.
The Gaussianization layers are based on optimization and fixed-point iterations, which can be compu- tationally expensive. To study possible simplifications, we estimated the forward and backward time for various combinations of sub-layers using 100 repeated experiments (Table LTb. The computation was performed using one Nvidia V100 GPU and an Intel Xeon Platinum 8276 CPU. We always turn on the ICA layer (together with whitening) since the ablation studies show that it contributes the most to performance increase (Table 2 and Table Combining results from those ablation studies, we can see that the option ICA+YJ strikes good balance between performance and computation time. It has a shorter runtime than LCA+tr and ICA+YJ+Lamberr At the same time, it gives comparable and sometimes the best results. On the other hand, the additional computation time from Gaussianization layers is well justified by the performance increase, especially if the physics simulation much more time-consuming. For example, the eikonal solver for the experimental setup in our study has a forward runtime of 0.293 seconds and a backward runtime of 5.933+0.155 151 seconds, respectively. In addition, our implementation can also be greatly improved by more efficient utilization of GPUs and rewriting bottleneck components using reghoperformanc codes.
Table 11: Forward and backward computation time of various combinations of Gaussianization sublayers. The times were computed using tensors of the same shape as latent tensors for Glow and StyleGAN2, respectively. Each computation time was estimated by 100 repeated experiments. We always turn on the whitening layer and the standardization layer.
(Note: the time is measured in seconds)
ICA
ICA + YJ
ICA + Lambert ICA + YJ+ Lambert
Forward (Glow tensor)
0.030
0.286
1,911 +0,33
2.022++0.04
Backward (Glow tensor)
0.017=386-4
0.022
0.751
0.
Forward (StyleGAN2 tensor)
0.033.13.ee4
0.280=
1.068
1.212
Backward (StyleGAN2 tensor)
0.027+226-4
0.037
1.077
0.964
We also provide insights into why Gaussianization layers perform better than other methods. First. the spherical constraint only relies on the geometric property of the Gaussian typical set (App. D However, as shown in Fig. it insufficient to only constrain the norm of latent vectors in inversion. Gaussianization is necessary to ensure inverted results are in the range of DGMs. Second, the orthogonal reparameterization generally underperforms the Gaussianization layers. Our interpretation is as follows: rather than actively destroying latent tensor patterns like the Gaussianization layers do, orthogonal reparameterization cannot prevent permuting the fixed typical Gaussian latent tensors into with spatial patterns. In addition, we have shown in Fig. 10 that the whitening transformation alone is ineffective in keeping the DGM outputs within the range. We thus interpret that reducing higher-order dependencies is essential. Therefore, the noise regularization and the whitening layers yield less satisfying results than the Gaussianization layers.
We believe this study benefit both the broader scientific community and the general public. However, we point out that the MRI and eikonal tomography experiments are purely synthetic and numerical. They do not fully reflect realistic medical imaging configurations. One should be cautious about applying our techniques to real data and interpreting the results.
33


Reviews:
[{"review_id": "qJdLALlesF", "reviewer": "Reviewer_VNJA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries solving inverse problems using generative models as regularizers. It is known that when doing MAP / Maximum Likelihood optimization, the optimization variable deviates from its expected statistics, which is standard Gaussian in most cases. \n\nThis paper proposes an algorithm that can enforce Gaussian statistics on the latent variable $z$ during optimization. This is done by adding a KL divergence regularizer to the optimization objective, where the KL is between the distribution of $z$ and a standard Gaussian. The authors then decompose the KL divergence as a sum of two non-negative terms, and minimization is done by alternating ICA / whitening on the two terms in the decomposition.\n\nThe algorithm is novel and the experiments are interesting, although I have some concerns about the baselines used for comparison.", "summary_of_the_review": "The algorithm is novel. The experiments are sort of convincing, but have clear issues. I am happy to raise my score if the authors can compare to the additional baselines I mentioned.", "strength_and_weaknesses": "**Strengths**:\n\n- I think the proposed solution is very practically reasonable. The algorithm requires minimal ``training'' to force Gaussian statistics on the solution. Previous works have considered using the vanilla $\\ell_2$ norm of $z$ (as in Bora et al 2017), while algorithms like PULSE[3] forcibly project to the Gaussian sphere. This solution is much better.\n\n- The decomposition in Eqn 6 is quite clever, as is the algorithm for iteratively minimizing the two terms in the decomposition -- the ICA keeps the LHS of Eqn 6 constant and increases the second term in the RHS, which implies the first term in the RHS must decrease; the next step is a whitening step that forces the second term to decrease while leaving the first term constant.\n\n- The experiments look really good, but it's hard to judge because the baselines considered are not the strongest. \n\n**Weaknesses**:\n\n- The motivation behind this problem is that an optimization problem as in Eqn (3) is unstable to choice of $\\beta$, if the regularizer is $\\beta || z ||^2$, as $z$ is supposed to be Gaussian distributed. But this doesn't make much sense in theory, as eqn (3) solves for the MAP in $z$, [which is not the same as searching for the MAP in $g(z)$ space, since $- \\log p(G(z)) = ||z||^2 - \\log det( \\nabla_z G(z) ) $]. This inconsistency in performing MAP in $G(z)$ or $z$ space could be the reason for the artefacts, as opposed to issues in optimization / distribution of the latent vectors.\n\n- An additional point about MAP in $z$ space: the phenomena described in this paper can also be circumvented when using GLOW by using something like Langevin dynamics to approximate Posterior Sampling [1]. In the case of StyleGAN, score-guided optimization[2] has been useful. In both cases the statistics of $z$ are respected. Your argument would be much stronger if you can offer a direct advantage (methodologically or experimentally) over these techniques. Does your optimization offer advantages over these?\n\n- The related work section is not very accurate -- score models can be used for non-linear inverse problems as well. The cited references [Jalal et al, Song et al] considered compressed sensing MRI and CT, which are linear inverse problems. I am not familiar with work questioning their applicability to non-linear inverse problems.\n \n- The baselines used for comparison are very weak, especially in Fig 4. To clarify, I'm not blaming the authors, it's well known that inverting a StyleGAN is empirically challenging. Comparing to compressed sensing experiments in [2] on the celebA dataset + StyleGAN would be a much more convincing and necessary experiment. (code and models: https://github.com/giannisdaras/sgilo )\n\n- The optimization procedure makes sense, but is there any guarantee that ICA and whitening leads to a strict decrease? I don't know what the theoretical guarantees for ICA are, and it would be helpful if you could add some references in the appendix.\n\n- Figure 5 has clear artefacts for the proposed method. What's going on? Can you provide a comparison to Langevin dynamics[1] on the same experiment?\n\n**Clarifications**:\n\n- The experiments section is a little confusing -- why do you have separate experiments for the Orthogonal Reparameterization and Gaussianization layers? I thought they're two components of one algorithm? What are these things in relation to Figure 2?\n\n-\n\n**References**:\n\n[1] Jalal, Ajil, Sushrut Karmalkar, Alex Dimakis, and Eric Price. \"Instance-Optimal Compressed Sensing via Posterior Sampling.\" In International Conference on Machine Learning, pp. 4709-4720. PMLR, 2021.\n\n[2] Daras, Giannis, Yuval Dagan, Alex Dimakis, and Constantinos Daskalakis. \"Score-Guided Intermediate Level Optimization: Fast Langevin Mixing for Inverse Problems.\" In International Conference on Machine Learning, pp. 4722-4753. PMLR, 2022.\n\n[3] Menon, Sachit, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. \"Pulse: Self-supervised photo upsampling via latent space exploration of generative models.\" In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 2437-2445. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "joTL5YeLseT", "reviewer": "Reviewer_NCs6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Recent work has shown how deep generative models (such as GANs or VAEs) can be used to regularize inverse problems by enabling optimization in the latent space of a pre-trained generative model—with a regularisation term to encourage latent code gaussianity. However, low-fidelity solutions whose latent codes strongly deviate from gaussian still occur. To address this issue, the authors propose a new approach: (i) perform optimization on unconstrained latent code, (ii) leverage a novel gaussianization layer to map the unconstrained latents to gaussian ones, which are then input to the generative model.\n\nThe authors apply their approach to three different inverse problems (compressive-sensing MRI, image deblurring, and eikonal tomography) and compare their approach to different methods for inverse problems (including methods that use generative models but different strategies to enforce gaussian latent codes). Across the three applications, the proposed approach achieves state-of-the-art performances.", "summary_of_the_review": "The authors introduce a new strategy for solving inverse problems using generative models by introducing a new layer enabling latent code gaussianization. This is potentially an important contribution for inverse problems, generative models and deep variational inference. The authors could improve paper readability providing a guiding example (see comments above) and release code for the proposed layers for broader impact. If the paper is accepted, the authors should provide code to reproduce their experiments.", "strength_and_weaknesses": "Strengths:\n* The new Gaussianization layer is a novel technical contribution, which combines several sublayers (ICA layer, power transform layer and lambert layer). This layer and its component can be applied to various deep generative models, inverse problems and deep variational inference models, thus being of potential interest to a broad community.\n* Results are convincing. Additional experiment/results are provided in the supplement. The authors also considered an ablation study to investigate which parts of the proposed Gaussianization layers were responsible for the observed performance improvement.\n\nWeakness:\n* I think the presentation of the paper could be improved. The authors may consider discussing a specific example when introducing inverse problems, regularization with generative models and the gaussianization layer. The deblurring example would be the easiest for this (the authors may also consider discussing this as the first experiment for consistency).\n* The authors do not comment on (future) code availability. I think that including Pytorch/Tensorflow implementations of the proposed Gaussian layer and its subparts would make this work more impactful. If the paper is accepted, the code to reproduce experiments should also be released.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aPTRhtTQw2s", "reviewer": "Reviewer_YtuR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper improves the deep generative models used for constraining solution spaces in inverse problems by making distributions of latent variables obey a Gaussian. Deep generative models such as StyleGAN2 and Glow are widely employed for the regularizes in inverse problems. Given latent variables that obey a Gaussian, these generative models generate realistic images. The authors point out that the latent variables deviate from the desired Gaussian distribution during inversion lead to poor results, and the proposed method reparameterizes and Gaussianizes the latent variables during the optimization for solving inverse problems. The proposed method was applied to compressive sensing MRI, image deblurring and eikoonal traveltime tomography, and the experimental results show the proposed method works well.", "summary_of_the_review": "The reviewer recommends to accept this paper because of the strong points described above.", "strength_and_weaknesses": "Strong points:\nThe authors point out that when the latent variables of deep generative models deviate from the desired Gaussian distribution, the solutions of inverse problems would be poor, and propose a method that makes the latent variables obey a Gaussian distribution. Using ICA, Yeo-Johnsonn transformation, and Lambert WxFx transformationo, the methd improves the independency, skewness, and heavy-tailedness of the distributions. The method is novel and the experimental results show that the proposed method can constrain the solutions more correctly. \n\nWeak points:\nThe algorithm for the update of v is not clearly described in the text. Appendix F describes the gradient computation of the optimization-based differentiable layers but the reviewer believes the contents shown in Appendix F should be described in the text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yn5IqJ2oBqM", "reviewer": "Reviewer_k6hw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to reparameterize and gaussianize the latent tensors for deep generative models such as StyleGAN2 and Glow. This idea was evaluated in a number of inverse problems such as compressive-sensing MRI, image deblurring, and eikonal tomography. \n\n", "summary_of_the_review": "This work introduced a Gaussianization layer for image generation models and then used it for solving a few inverse problems. This work was compared with the works not using Gaussianization. However, using Gaussianization for image generation is not new as discussed in the review and some of them look critical. Moreover, it is unclear if the intended Gaussianization really happened in all problems - no quantitative results for Gaussianization. Thus, in its current form, without justifying its novelty over (C Meng et al., AIStats 2020) and other works using the idea of Gaussianization, it will be very hard not to recommend rejection.", "strength_and_weaknesses": "Strength:\n- Unless this work clearly resolved the first issue in the weakness, it is hard to write down any strength since it is unclear if those strengths are belonging to this work or other works.\n\nWeakness:\n- This work looks very similar to the following prior work: C Meng, Y Song, J Song, S Ermon, Gaussianization Flows, AIStats 2020. The key idea of this work is to reparameterize latent vectors using learnable orthogonal matrices (Cayley parameterization) while the work of Meng et al. also used \"trainable orthogonal matrix layer.\" This is really critical in my assessment for this work and it is really unfortunate that this work missed this prior work. Thorough comparison with this prior work seems critical for this work to be justified as a new work.\n- It is unclear if all comparisons in this work were proper. For example, (Wulff & Torralba 2020) also investigated on Gaussianization in the latent space for StyleGANv2 and this work should be compared with the proposed method with StyleGANv2 in terms of performance and the level of Gaussianization. \n- It is unclear if the proposed methods worked as designed or not. All comparisons show that the proposed methods outperformed other selected methods, but there is no investigation on how much Gaussianization was achieved with the proposed methods. It will be helpful for many readers to see some concrete quantitative results on Gaussianization itself.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster