Regularized Training of Intermediate Layers for Generative Models for Inverse Problems
Authors: Sean Gunn
Jorio Cocola
PAul HAnd
Venue: Accepted by TMLR ()

Abstract: Generative Adversarial Networks (GANs) have been shown to be powerful and flexible priors when solving inverse problems. One challenge of using them is overcoming representation error, the fundamental limitation of the network in representing any particular signal. Recently, multiple proposed inversion algorithms reduce representation error by optimizing over intermediate layer representations. These methods are typically applied to generative models that were trained agnostic of the downstream inversion algorithm. In our work, we introduce a principle that if a generative model is intended for inversion using an algorithm based on optimization of intermediate layers, it should be trained in a way that regularizes those intermediate layers. We instantiate this principle for two notable recent inversion algorithms: Intermediate Layer Optimization and the Multi-Code GAN prior. For both of these inversion algorithms, we introduce a new regularized GAN training algorithm and demonstrate that the learned generative model results in lower reconstruction errors across a wide range of under sampling ratios when solving compressed sensing, inpainting, and super-resolution problems.

Published in Transactions on Machine Learning Research (01/2023)
Regularized Training of Intermediate Layers for Generative Models for Inverse Problems
Sean Gunn
gum..s@hortheasteem.ell
Khoury College of Computer Sciences Northeastern University
Jorio Cocola
Soncola@@sess.harveen edu
Instibute for Applied Computational Science Horvard University
Paul Hand
p.landd@horthheasseey edu
Department of Mathematics and Khoury College of Computer Sciences Northeastern University
Reviewed on OpenReview: /ttps: openreviee. net/ forum? id- cKsKXR28cG
Abstract
Generative Adversarial Networks (GANs) have been shown to be powerful and flexible priors when solving inverse problems. One challenge of using them overcoming representation error, the fundamental limitation of the network in representing any particular signal. Recently, multiple proposed inversion algorithms reduce representation error by optimizing over intermediate layer representations. These methods are typically applied to generative models that were trained agnostic of the downstream inversion algorithm. In our work, we introduce a principle that if a generalive model is intended for inversion using an algorithm based on the optimization of intermediate layers, it should be trained a way that regularizes those intormediate layers. We instantiate this principle for two notable recent inversion algorithms: Intermediate Layer Optimization and the Multi-Code GAN prior. For both of these inversion algorithms, we introduce demonstrate that the learned generative model results in lower reconstruction errors across a wide range of under-sampling ratios when solving compressed sensing, inpainting, and super-resolution problems.
regularized GAN training algorithm and
1
Introduction
The task in an inverse problem is to estimate an unknown signal given (possibly noisy) set of measurements of that signal. In practice, inverse problems are often ill-posed, which often require the incorporation of prior information about the target signal to recover reasonable estimate of it.
Deep generative models have demonstrated remarkable performance when used as priors for solving inverse problems (Bora et al. 2017 Athar et al. 2018 Hand et al. 2018 Menon al. 2020 Mardani et al. 2018 Mosser et al. 2020 Pan et al. 2021). Typically, generative modeling- based approach for inverse problems has two phases: a training phase and an pvveriion/deplyymee
phase. In the first phase, generative network is trained on a dataset of images. After training, the network parameters are typically fixed and an optimization problem involving the known forward model is solved to estimate an unknown signal of interest. An advantage of this class of methods is that the prior can be trained entirely in an unsupervised manner and without previous knowledge of the specific inverse problem that needs be solved downstream. This allow the trained network to be used for variety of inverse problems.

---

Published in Transactions on Machine Learning Research (01/2023)
Some generative networks explicitly map a low-dimensional imput to high-dimensional signal space. The outputs of the generative network, therefore, have a low intrinsic dimensionality. While this is useful for regularizing the inverse problem, it can limit the expressivity of the network and lead to representation error, the error between a target signal and the closest signal in the range of the generative network.
Several recent methods have attempted to reduce representation error by introducing optimization algorithms that enlarge the search space of the inversion algorithm. These include methods that optimize both over the input of the network and over the activations of one more intermediate layers. Examples of this class
of inversion methods include Intermediate Layer Optimization (ILO) (Daras et al.) 2021), GAN Surgery (Smedenark-Margguliss ct al. 2021), and the Multi-code GAN Prior (mGANprior) (Gu et al. 2020). These algorithms were applied to state-of-the--rt off-the-shelf GANs, such as PGGAN and StyleGAN Karras lt al. 2018 2020), which were trained agnostic to the specific optimization algorithm that would be used for inversion. Consequently, the inversion algorithms optimize over a region of the space of intermediate presentations that were not directly regularized during training. This suggests that it may be possible improve the reconstruction performance of these algorithms by explicitly regularizing the intermediale layers during the training phase. In this paper, we confirm this hypothesis.
We consider the problem of training a GAN that will be used for solving inverse problems via an inversion algorithm that optimizes over intermediate layers. We put forward the following principle termed Regularized Training of Intermediate Lay- ers (RTIL):
Vanilla training of GAN
Deployment Phase
Algorithm that optimizes over intermediate layers
20
x
Algorithm that optimizes over intermediate layers
Training Phase
If a GAN's intermediate layers are optimized during inversion, they should be regularized during training.
20
We use this principle to derive new train- ing algorithms for GANs that are intended for solving inverse problems. For ILO and mGANprior, we introduce a new training algorithm for StyleGAN2 and PGGAN, respectively. For these new trained GANs, we achieve improved reconstruction qual- ity relative to the corresponding GANs trained without the principle. We note that this paper is not about regularizing GANs in order to stabilize training and aiding the convergence of gradient descent (as in Gulrajani et al. (2017a];: Meschoder et al. 2018))). Rather, those regularization methods can be used in conjunction with RTIL, to improve upon approximate learned inference for a specific algorithm used in solving inverse problems.
Regularized training of Intermediale Layers
3
Labont vartalbles
Optimized variables
x
GAN layers
P2
Figure The workflow for using Regularized Training of Interme- diate Layers (RTIL) to develop new training algorithms of GANs that will be used for inversion. See the text for details.
Our proposed principle may lead to the following workflow for solving inverse problems with GANs. First, train a generative network with latent variable 2p, sampled from a latent distribution Pzo and outputting signals x from the target distribution (Figure Step 1). Sccond, explore various inversion algorithms, including some that optimize over intermediate layers by introducing an additional optimization variable ø1 (Figure Step 2). Algorithms of this type include ILO and mGANprior. Third, if such algorithm provides competitive performance for inversion, then use RTIL to devise a new GAN training algorithm. This can be achieved by introducing a new latent variable =1 Psz where Pz, is an appropriate probability distribution (Figure Step 3). Finally, during deployment, use this trained generative network for inverse problems via the selected inversion algorithm (Figure Step 4).
This workflow demonstrates a way to use recently introduced inversion algorithms in order to inspire new training algorithms for GANs. It provides additional ways of training GANs knowing that they will be used
2

---

Published in Transactions on Machine Learning Research (01/2023)
for inversion, and it provides way to enable some empirically successful inversion algorithms operating in more principled manner by ensuring they are searching over a space of parameters that have been suitably regularized.
The contributions of this paper are the following.
We introduce Regularized Training of Intermediate Layers, a principle for training deep generative networks that are intended to be used for inverse problems when solved by an algorithm that optimizes over intermediate layer representations.
In the case of Intermediate Layer Optimization, we use our principle to devise a novel GAN training algorithm. With the resulting trained GAN, we demonstrate lower reconstruction errors (compared to GAN training without the principle) for compressed sensing, inpainting, and super resolution over a wide range of under sampling ratio.
We show the versatility of the method by repeating the same contribution in the case of the Multi-Code GAN Prior.
We illustrate the benefits of compressed sensing with RTIL by theoretically showing that a model trained without regularizing intermediate layers has a strictly larger reconstruction error than a model where the intermediate layers were regularized during training. For simplicity, this result is established in the case of two layer linear neural network trained in a supervised setting.
1.1 Related Works
In recent years, various approaches have been proposed for dealing with the representation error when solving inverse problems with learned generative prior. As described above, one class of approaches focuses on enlarging the latent space dimension, either at training at inversion time (Athar et al. 2018 Dhar et al.
2018 Hussein et al. 2020). On the other hand, another notable class of approaches recently put forward is based on flow- bas-based invertible neural networks. These generative networks have invertible architectures and a latent space with the same size of the image space, and thus have zero representation error (Ardizzone 2019 Asim et al.] 2020; Ma Le Dimet 2009) Kelkar et al., 2021; Shamshad et al. 2019 Whang 2021a Helminger el al., 2021). Both these type of approaches atlempt to limit. the representation
le al.
et al.
error during training or inversion.
Similarly to flow-based models, score networks and subsequent variants (Song & Ermon 2019
2020 Song
et al. 2020) have support on the entire signal space, allow for conditional sampling and likelihood estimationn. and have been object of recent interests for their use in inverse problems (Ramzi et al. 2020 Jalal et al. 2021a b).
The increased representation power of the above-mentiones generative networks, usually comes at a price of increased computational cost, both during the training and the inversion phase. In contrast, our proposed principle leads to training algorithms that have essentially the same computational cost the standard ones, also leaving the cost of the inversion algorithms the same.
2 Generative Models and Optimization Algorithms for Inverse Problems
.11 Background
We consider the problem of training GAN for the use of a prior for solving inverse problems. We consider a GAN G which maps a latent space RT to an image space RT*,, where no < na- In this paper, we focus on the linear imaging inverse problems of compressed sensing, impainting, and super resolution, though our proposed method also applies to nonlinear inverse problems such as phase retrieval, and inversion problems about non-image signals. We consider the general lincar inverse problem of recovering an image E R from set of linear measurements y E Rm given by y = A(ar), where A : Rea - Rmm is a forward linear measureentt operator. We study only the noiseless case, but method easily extends to the case with measurement noise. In this paper, we study measurement operators A of the following form:
3

---

Published in Transactions on Machine Learning Research (01/2023)
Compressed Sensing: A = A E Rmxna, A is random matrix that samples from known distribution with m < ndd
Inpainting: A= = M E Ruxns M is masking matrix with binary entries.
:upot-Resolution: A mmy R where Smy is the downsampling operator with downsampling factor m4.
An estimate of a can be recovered by finding the image in the range of G that is most consistent with the measurements y in the following sense, as introduced in (Bora et al. 2017 First, solve
20 arg min y A(G(o))))
(1)
then, the estimate of To is given by G(zo).
As mentioned in the introduction, a difficulty this optimization approach is that the estimated images are constrained live within the range of G, which is no-dimensional manifold in R"s, Most images 2 will not live exactly in this rango, and thus the method is limited by the representation error ming, ||æg G(z))ll- In the next sections, we review two recent algorithms for miligaling representation error during inversion. For ease of exposition, we will discuss the case where only one intermediate layer representation is optimized. We write G = go where 9o : I + R. and RT + RM,
.2 Intermediate Layer Optimization (ILO)
Given a trained generative model G 91 go, Intermediate Layer Optimization (ILO) extends the range of the generative model by sequentially optimizing over each layer of the network, as demonstrated in Algorithm The initial step begins exactly as in (Bora et al. 2017] by optimizing over the input vector Zg E R The solution is obtained in line 3, by initializing a 2o ~ NO, Ino): then optimizing the loss with gradient descent. After the solution &g is obtained, the algorithm scarches for a perturbation 21 of go that further minimizes the reconstruction error (line 4). The final approximation of the target image æ is given by gı + 9o(2))) As the authors point out, there are multiple ways where IT.O can be regularized, including by an L1 penalty in the intermediate representation, or via early stopping. For the present paper, this was the method used by the publicly available code from the authors. Throughout this paper, we use the code provided by the authors when solving IT.O.
early stopping,
..3 Multi-Code GAN (mGanPrior)
Multi-Code GAN Prior is an inversion method that simultancously optimizes over multiple latent codes and composes their corresponding intermediate fea- ture maps with adaptive channel importance. This effectively extends the expressivity of the network by giving it higher dimensional input space and addi- tional parameters to control the importance of each channels in the intermediate layer representation.
Algorithm 1 Intermediate Layer Optimization (ILO)
for Compressed Sensing (Daras el. al. 2021
1: Input: G= 91 o 9o, measurement matrix A E
Rmxna compressed measurements y.
2: Output: estimated image a
3: 20 = arg min - y (90(())))) Initialize at Zu
Assume we are given a pre-trained generative model G == 91 o 9o: where the output of the first layer 9o has dimension H1 x W1 x Ci with C1 being the number of channels. Furthermore, chose N latent codes 1 E g and channel importance {a* j e R.. Then the Multi-Code GAN Prior extended architecture computes 91 1E @ a*) where {0))) @ a² )aye =
20
(0, Ing)
4: 2 = arg min 1 - A21 + go (o())) Initialize at
21 ~ NN
5: Return: & 91 + (E
{a*) is channel-wise multiplication, ú,j are spatial location, and e is the
channel index.
During the inversion phase, the Multi-Code GAN Prior method optimizes over both the latent vectors {= and the channel importance {a 1= (Algorithm 2
4

---

Published in Transactions on Machine Learning Research (01/2023)
Ground Truth
Ours
ILO
Ground Truth
Ours
mGANprior
SSIM - 857
SSIM-, 786
SSIM- 938
SSIM-.874
XCO
KCG
M
SSIM=,865
S51M=.783
SSIM=,894
SSIM=.832
(a) Comparison between ILO-RTIL (ours) and ILO for compressed seusing for 3% of measurements.
(b) Comparison between mCAANrior-RRIIV (ours) and mCANNrior for compressed sensing for 5% of measurements.
Figure 2: Results the left compare ILO-RTIL to ILO and then the right mGANprior-RTI RTIL to mGANprior.
3 RTIL
In this section, we present how the principle of Regularized Training of Intermediate Layers (RTIL) inspires training algorithms for GANs that are intended for inversion by Intermediate Layer Optimization (Section 3 and mGAN- prior (Section 3 2
Algorithm Multi-Code GAN (mGANNrior)
Gu et al.
22200
Input: Trained network G 91 9u. latent codes { E R {a* fk=1 E I measurement matrix
We consider the case of a practitioner having chosen, after some initial exploration, a base generative network G for use as prior solving inverse problems. For simplicity, in this section will consider G 91 9o where Oo R -> R and _ RR The input latent vectors of the network G are sampled from Pso (e.g. Pso = NO In)) and 0 is the set of trained parameters.
A E Rmxxa compressed measurements y.
2: Output: estimated image & Inilialize { ~ Pz: {a Jk=1 ~ Po
V.
ε arg min 1yy Agi (E g0(2)) o a²)ll
(=0))
Return: a 91 (Σ 90(2)) ④aa)
We assume, furthermore, that the practitioner has selected an inversion algorithm that optimizes over the latent variable zo and the intermediate layer between go and 9i of G. Optimizing over the intermediate layer corresponds to introducing free variable 21 between go and 91 (Figure
The RTII. principle states that if one intends to solve inverse problems by means of intermediate layer optimization algorithms, then intermediate layers optimized over during inversion should be regularized during training.
This principle can be used to design new training algorithm the following manner. We identify the additional free variable, 21, used for optimization over the intermediate layer, consider as a latent variable of the generative model, and provide it with a simple distribution Pzz- For example, this could result in generative model R x R Ra,, such that G(20- 21) g1 (i + 90(2))), but could have alternative functional forms. This generative model reduces the base generative model G R - R if 2 is suitably chosen, for example if 2 = 0.
5

---

Published in Transactions on Machine Learning Research (01/2023)
The introduction of the latent variable 21 explicitly increases the dimensionality of the latent space. In practice, training latent variable models with high-dimensional latent spaces can be challenging and require careful regularization (Athar et al. 2018). We address this difficulty by concurrentyy training the lower and higher dimensional models Gg and Gø, which share trainable weights 0 (Figure) 3
We train Gø and Go via the following minimax formulation Goodfellow et al. 2014
min max Ex-pe log (2) + Ezaoppo #1mPa1
[Alog(1 - De((Go(z)) + (1 A) log(1 - (e(G =1)]],
(2)
where Dy: Th + R is the discriminative network and E [0, 1] is hyperparameter. This method could
extended alternative GAN and non-GAN formulations (Arjovsky al. 2017 Gulrajani et al., 2017b
Bojanowski et al. 2018
Once G is Irained, a practitioner could solve an inverse problem with forward operator by solving
E0, 21 arg min I1yo - A(G((zo. 21))1l:
20.2
using the selected optimization algorithm, resulting in G( 21) the estimate of the signal.
In this section, considered the case where only intermediate layer optimized. The proposed method can directly extend to the case where multiple layers are optimized.
We will next describe the details of the application of RTIL in the case of Interme- diate Layer Optimization and the Multi- Code GAN Prior.
Inversion
RTIL
90
91
90
91
P=0
3 .1 RTIL for ILO
In the case of Intermediate Layer Opti- mization (ILO), we now present how to use RTIL to design a GAN training al- gorithm. Consider the base generative network G(zo) = 91 (9o(zo)),
Latent variables
20
99
Optimized variables
Parameter sharing
be used for inversion with Intermodiate Layer Op- timization. II.0 (Algorithm [ extends the range of the network by optimizing gz(z1 + (o(2o)) over latent variables and z1- Consequently, we consider the higher dimensional generative model
Visual representation of the RTIL principle. An in- version method that optimizes over intermediate layers informs the training of a family of generative networks adapted to ILO inversion method.
Figure 3:
G(Z0, 21) = 91 + (o(o))), 20 ~ ~(0, 21 ~ ~
where o2 is hyperparameter. Then simultaneously the lower dimensional model G(zz) and higher dimensional model G(z0, =1) are trained with the min max formulation above. Note zo ~ Pso and ø1 ~ Pz, where Figure depicts this process.
3 .2 RTIL for mGANprior
In the case of Multi-Code GAN Prior method, we now present how to RTIL design GAN training algorithm. Consider the base generative network G to be used for inversion with the Multi-Code GAN Prior method. The mGANprior Algorithm (Algorithm 2 extends the range of the network by optimizing O a² over latent variables 2g and 21. Consequently, training the higher dimensional model

---

Published in Transactions on Machine Learning Research (01/2023)
yields
Glzg...... =0 a ,.... a
".......a"" 91 (0(20) a²), z ~ ~N110,0, Ino): Pa Dir
where each vector a is taken to be a {a h where 1 is the vector of all ones and {a'}x is the k-th entry of a' E TRN. The vector a' is sampled from Par ~ (ir where (1) is the flat Dirichlet distribution, i.e. the uniform distribution over the (N -)-mmmensinn simplex. Note this leads to each channel being weighted equally during training. The training process is depicted in the appendix, Figure
4
Experiments
Compressed Sensing with Circulant Matrices
Inpainting
Super-Resolution
38
37.5
34
35.0
32
32.5
(d8)
30.0
PSAR (dB)
28
30
26
20.0
22
22
20
30
0.3 Downsample Factor
Percentage of Measurements
Percentage of Observed Pixels
RTIL-ILO
ILO
Figure 4: Performance of ILO-RTIL and vanilla trained ILO for Compressed sensing, inpainting, and super resolution for various under-sampii samplisg ratios. ILO-RTIL increases performances or ties for each mader-sampling ratio with respect to PSNR across each of the inverse problems compared to ILO, The vertical bars indicate 95% confidence intervals.
We observe that Regularized Training of Intermediate Layers is not lied to any specific architecture training procedure, and while, until this point, have only presented it on two-layer networks for easiness of exposition, in this section demonstrate its successful application two state-of-the-art architectures on different imaging recovery problems. Specifically, we conduct extensive experiments comparing RTIL versus vanilla training for compressed sensing, inpainting, and super-resolution, and for two different inverse methods Multi-Code GAN Prior (Section 2 3) and Intermediate Layer Optimization (ILO) (Section 2 .2) to demonstrate the effectiveness of our method. The generative model architecture used for the mGANprior is PGGAN [Karras al. 2018 and for ILO is styleGAN-2 (Karras et al., 2020). All models were trained on FFHQ data set (Karras et al. 2019) and tested on CelebA-HQ dataset (Karras et al. 2018), at image size of 256x256x3. The choice of architectures were based the experimental section of the original ILO (Daras ct al. 2021 and mGANprior (Gu et al. 2020) papers. Refer to the appendix for the architecture details for training networks using RTILL, as well as hyperparameters chosen for inversion methods.
For all experiments, compressed sensing use partial circulant measurement matrices with random signs Daras et al. (2021), inpainting the pixels are missing at random, and downsample factor corresponds how much the height and width of the original image was reduced All experiments were trained with À equation The code is hosted GitHub repository http:::/gitubb.omm/ g33san RRIIL
4 .1 Results ILO-RTIL
Results in this section correspond Figure 4 where the results are averaged over 100 images randomly sampled from CelebA-HQ. Note, the hyperparameter set d² during training for all experiments.
7

---

Published in Transactions on Machine Learning Research (01/2023)
Compressed Sensing ILO-RTIL demonstrates increase in reconstruction performance across each under- sampling regime. The largest increase in performance with respect to PSNR occur at 25% (1.28 dB), 15% (1.59 dB), and 10% (1.3 dB's) measurements. For qualitative results, please refer to Figure ((a) Inpainting- ILO-RTIL demonstrates in an increase in reconstruction performance with 4 out of the sampling regimes. There is an increase in PSNR at 50% (1.88 dB), 25% (1.59 dB's), 15% (.94 dB's), and 5% (.29 db) of observed pixels. At 1% of observed pixels, there is a decrease performance in reconstruction bars overlap each other, indicating there is significant improvement. For qualitative results, please refer to Figure 16 Reger--esolttion ILO-RTIL shows slight increase in performance over the entire measurement regime compared ILO, the most significant occurs at downsampling factor where on average the increase in reconstruction is .77 dB's and the 95% error bars are separated. All other sampling regime achieves comparable performance between ILO--TIL and IT.O. For qualitative results, please refer to the appendix
.5 dB's, but the error
Figure
Compressed Sensing with Circulant Matrices
Inpainting
Super Resoluttn
30
30
28
+
PSNR (dB)
PSNR (d8)
PSNR (dB)
26
24
22
25
22
24
0
20
40
60
0
10
20
30
40
50
0.1
0.2
0.3
0.4
0.5
Percentage of Measurements
Percentage of Observed Pixels
Downsample Factor
RTIL-mGANprior
mGANprior
Figure 5: Performance of mGANppiir-RTIL and vanilla trained mGANprior for compressed sensing, inpainting, and super resolution for various under-sampling ratios. mGANprior-RIILL increases performances over vanilla mGANprior with respect to PSNR over each under-sampling ratio, except for super-resolution problems at low under-sampling ratio's. The vertical bars indicate 95% confidence intervals.
4.2 2 Results RTIL-Multi-Com
Results in this section correspond to Figure 15 where the results are average over 100 images randomly sampled from CelebA-HQ and all experiments use N 20 latent codes. Compressed Sensing mGANprior- RTIL has significant. improvement in reconstruction performance over mGANprior. For each under-sampling ratio, our method increases performance by at least 1.6 dB's, which clearly separates the error bars. On the other hand, at 1% of measurements where both methods achieve the similar performance, which is seen by overlapping eтrоr bars. For qualitative results, refer to Figure 2(b) Inpainting- IGGNpprio-RTIH demonstrates a significant improvement in reconstructing over mGANprior clearly being able the error bars Our method yields an increase performance with respect to PSNR of 2.5 dB's at 50%, 2.23 dB's 30%, 1.99 dB's 20%, and 1.58 dB's at 10% observed pixels. For 1% of observed pixels, these both methods yield similar results. Lastly, qualitative results can be seen on Figure 17 Reppe--Reoolu mGANppior-RRII show substantial improvement with image reconstruction at downsampling factor of and 1. increase of 2.25 dB's and 1.51 dB's respectively. However, at T6 down-sampling factor mGANprior outperforms mGANppiir-RTTT. slightly, but within the error bars. For qualitative results, please refer the appendix Figure 19
..3 Ablation Study
This section provides two ablations studics, comparing vanilla training versus RTIL for cach intermodiate layer trained using LO-RTTIL and mGANprior-RTI Each experiment was tested images sampled randomly from Ce..bA-HQ.
8

---

Published in Transactions on Machine Learning Research (01/2023)
4 .3.1 ILO-RTIL
P recall All results refer to Figure
Figure 3 and Algorithm for no- tation, 20 refers to optimizing over the initial latent vector (Bora et al. 2017 denotes sequentially opti- mizing over the first two layers, and so until representation eгrоr 24. Re- for to the appendix for number of iterations per intermediate layer.
Compressed Sensing with Circulant Matrices
Inpainting
35
PSAR (dB)
-30
32
30
255
Compressed Sensing Overall, opti- mizing up to 24 achieves the best performance for IL.--TIL and ILO, in most snder-sampling ratio's outperforms and at worst lies com- pared to other intermediate opti- mization layers. Comparing ILO- RTIL ILO, there an increase in reconstruction performance age across each under-sampling for of 1.41 dB's, .81 dB's, Z2 .4 dB's, and 2j .86 dB's. Inpaiting- Refer to appendix for inpainting results in Figure
20
20
25
30
Percentage Measurements
Percentage Observed Pixels
LO-TIL
ILO
Figure 6: Comparing compressed sensing and inpainting reconst performance between ILO-RTIL and ILO for a various number of intermediate layers.
Inpainting- Overall, ILO-RTIL soquentially optimizing up N4 performs the best overall, outperforming other intermediate representations 23, 21 at 50% observed pixels then achieving the slightly better performance at the under-sampling ratio's. Comparing ILO-RTIL ILO, there is increase in performance average between each experiment for 1.27 dB's, .55 dB's, .4 dB's, and z 1.27 dB's. Moreover, ILO-RTIL from 21 to 23 average there is increase of performance of .5 dB's, where for vanilla trained there is a decrease of 288 dB's.
.... mGANprior-R RTIL
Compressed Sensing- mGANpior- RTIL outperformed vanilla mGAN- pior in each optimization setting using N = {1, 10, 20), marginal at N = 1, but more notice- able at N (10,20}. Overall, N 20 achieves the best perfor- mance for both mGANprior and mGANprior-RTIL training across each under-sampling regime. As well as mGANprior-RTie RTIL increases performance on average across all the under-sampling experiments with N 10 compared to N 20 with mGANprior. Inpaiting- Refer to appendix for inpainting results in Figure Overall, N = 20 achieves the best performance for both vanilla and RTIL training across each sader-sampling regime. For mGANprior has substantial
Compressed with Circulant Matrices
Inpainting
30
30
28
26
PSNR (40)
244
22
20
0
20
40
60
80
100
0
10
20
30
40
50
Percentage Measurements
Percentage Observed Pixels
moAMtrror N-20
mGAb Rr-RI N-10
MGAN No1 MEAN-RTI Nm-1
moAMtrior N=20
mGAA--ITLL N-10
Figure 7: Comparing compressed sensing and inpainting reconstructing performance between mCGANpriorRRTIL and mGANprior for various number of latent codes

---

Published in Transactions on Machine Learning Research (01/2023)
increase in performance (1.83 dB's) between 10 latent codes and 20 average across each experiment. RTIL training increases performance not as much, but a notable amount with an increase of .93 dBs. However, mGANpprir-Rril with N = 10 outperforms mGANprior on average across all the under-sampling ratios of about .66 dBs.
5
Theoretical Model for Compressed Sensing with RTIL
In this section, we examine a simple theoretical model for compressive sensing with RTIL. Our objective is to shed some light on the reasons why RTIL may improve the performance of inversion algorithms that optimize over intermediate layers.
We begin with an informal discussion. Recall that for a base generative architecture G = I1 given by the composition of two computational modules IR + Rd1 and 91 : IR= + R ILO uses the extended generative network G R x R + R such that G( x1) = 91 (z1 + 90(2))).
We would like to compare the performance of an extended model @Van trained in a Shandard/vanille way and an extended model ÇRTTH trained using RTIL the principle. In the vanilla case, during training, the intermediate layer variable 21 is ignored and the network may be trained via the standard minimax formulation
min max Ee [log D=)+)] Exo-P=n [log(1 De(G. (zo. aa (e 0)]]
0
Using the RTIL. principle instead the network GRTIL may be trained with (taking x = 0 in 2 for simplicity)
min max Ex-pa [log De(x)) + EzoePao 21m
[log(1 De(G DaaGGrrll (zo, =1))]
(4)
6
For compressed measurements y with the sensing matrix A, then ILO would perform the following steps (see Algorithm
20 = arg min y AG(z' 0)12. initialize 2/ ~ NN(0, Ino):
2 CR=
(5)
1 arg min 11 A initialize z ~ (0)) E
where G = Gyan
G or G = RTI
Notice now, that when training Van using we compute Van 0) = gi (go(())) During training, the computational block 91 therefore only receives input vectors in the range of go. and not specified in the directions off the range of 9o- During inference, in the second step of though, compute Van (20, = 91 (u(()) + z Hence, ILO optimizes the variable z in directions for which the behavior of 9i may not be well specified, which could lead to large errors.
Contrary to vanilla training, when training GRTIL with RTIL we compute @RTIL (zo, 21) = 91 (90(20) + z1)- Hence, we expect the computational block J1 to learn meaningful outputs for inputs gu(zo) + =1 off the range of go and II.O to perform better.
In the next sections, formalize the above argument. To provide the most casily understandable context, we consider the case of generative model given by two-layer linear neural network. Further, we consider the case where the generative model is trained in a supervised manner, and we consider the regime of infinite training data. The supervised setting allows us to avoid the technicalitics of unsupervised adversarial training, and working the infinite data regime allows us to avoid statistical estimation errors.
5 .1 Problem Setup
We assume that the true signal distribution is given by x = G* x1) = WF + #1), where G* Ro x R + R 2g (O, Ino) and 2 (0, I,,) drawn independently, and where Wit E R and W Ra We moreover assume that no < ni < nd, and Wt. Wi are full rank. We furthermore assume that Wi is known, but that W. is unknown.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Let x² = G* (z). be unknown signal, where z (O and z ~ In)) We consider the problem of recovering x² given compressed linear measurements y Ax E R where S m < ndd
We assume that a practitioner has chosen a base generative model G R + R given by G 20 + G(zo) = Wi) Since this model has a nonzero representation error the practitioner will then consider the extended generative network (zz. =1) : 1o x R + R= given by G: 20, 21 + G(Zo. =1) = WwWW #o 21), and an inversion method that estimates x² as x² = ( 21) where (Eu, 21) are found using ILO
We are then interested in analyzing the effects of different training methods for T on the estimation of x* via the inversion method 5
5 .2 Training the Models
We will now compare the training of G with vanilla training and with RTIL. The model trained with vanilla training will be denoted by Van and its weight by WYan. The model trained with RTIL will be denoted by GRTIL and its weight by W.TTL
Training each model consists of estimating W/an and WRTHL under least squares loss. In the idcalized regime of infinite training data, vanilla training the generative network Van corresponds to ignoring the intermediate variables 21, and in particular estimating Wi by
W1Y Van E arg min Exg,2, ||G (z0, 21) - V
Van
(zo, VVa O).
(6)
W. ex
Since the objective is to solve compressed sensing with the intermediate layer optimization training GRTIL with RTIL instead corresponds to
WRRTL E arg min Ez..z. | 21) CRTIL (Zo, 21)1
Wi Xx
The next lemma characterizes the solutions of the optimization problems and
Lemma .1. Let W Van in then W1 Wi Wi WG. Moreover, there erists a unique WiITILL solution of given by W.RTH. Wr.
While the above lemma shows that W.an will equal Wf the range of W its behavior the space orthogonal to the range of Wi depends on how t is solved. To simplify the discussion below, we will consider W/Yam Van to be the minimum (Frobenius) norm solution of 6
5 . Compressed Sensing
We denote by (z)an, Yy)an the solutions of the minimization problems 5 using Van in place of G. The estimate of x² using the vanilla trained model is then x= VVa (2) Van = Van). Similarly, ( =RRTLL) are the solutions of using GRTIL place of G, so that the estimate of x² using the RTIL model is x² Van (=vam<2 (RTT 21 2RIL...
The following lemma quantifies the reconstruction errors made by using the two generative models.
Lemma 5 .2. Assume that Wi, Wi are full rank. Let W. be the minimum norm solution of and W. be the solution 7 Let A E Rmxna with i.idd. NO, 1) entries. Then with probabilily 1
E Ezg.2 [II'((z;,1)
G Van (z Van 2 Van
(=)an
max hEmnge(W;))
I1(Ina Pw;w;) W wrrll
(8)
and
Ex, =; [|IlG((T)) 21) 1 NTTIL RTIL x
(x
RTHH 112] =
(9)
The above results illustrate why training generative models using RTIL can enable better compressed sensing performance. In the simplified setting of two-layer línear neural network trained in supervised manner
11

---

Published in Transactions on Machine Learning Research (01/2023)
with known first layer and in the infinite data regime, that the generative model trained with vanilla training incurs error in the second layer's weights in the orthogonal complement of the range of the first layer. This results in an increase in the reconstruction error when solving compressed sensing.
6
Conclusion
We have introduced principle for training GANs that are intended to be used for solving inverse problems. That principle states that if the inversion algorithm optimizes over intermediale layers of the network, then during training the network should be regularized in those layers. We instantiate this principle for two recent and successful optimization algorithms, Intermediate Layer Optimization (Daras et al., (021) and the Multi-Code Prior (Gu et al. 2020). For both of these algorithms, we devise a new GAN training algorithm. Empirically, we show our trained GANs allow better reconstruction in compressed sensing, inpainting, and super-resolution across multiple under-sampling regimes when compared to GANs trained in vanilla manner. We note that our methodology only applies in the case of inversion methods that optimize over intermediate layers. However, there have been multiple competitive methods of this form published recently, each of which we show can benefit from this approach. Tools like those proposed in this paper, provided sufficient. computational resources, may allow these methods to be even more competitive in real-world application ciplications. verification at larger image sizes and on generative models trained specifically for certain applications, such MRT can benefit from the increase in performance.
7
Acknowledgements and Disclosure of Funding
PH acknowledges support from NSF Awards RMS-2053348,,
and DMS-2022205.
12

---

Published in Transactions on Machine Learning Research (01/2023)
References
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ulrrcch Köthe. Analyzing inverse problems with invertible neural networks. In International Conference on Learning Representations, 2019. URL https:
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 399-409. PMLR, 2020.
ShahRukh Athar, Evgeny Burnaev, and Victor Lempitsky. Latent convolutional models. ar preprint arXiv:1806.0622 062,1, 2018.
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative networks. In International Conference Machine Learning, pp. 600-609. PMLR, 2018.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alex Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2421 2432, PMLR, 2021. URL http:///rooceedings mlr wttppppss/vv///vprr.eea daras21a.htmm
Manik Dhar, Aditya Grover, and Stefano Ermon. Modeling sparse deviations for compressed sensing using generative models. In International Conference on Machine Learning, pp. 1214 1223. PMLR, 2018.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014. URL httpps//prooceednngs.. http:///www...aoeoo....co/pppp/e 5ca3e9b122 (a226112266118000060777attt-2333
ppff
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recogriition (CVPR), June 2020.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans, In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017a. URL httpp::/proceednng..eeeese http://ig....aurrooooco./pppp//p pdf
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In NIPS, 2017b.
Paul Hand, Oscar Leong, and Vlad Voroninski. Phase retricval under a generative prior. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL /ttpp::/pprocedings .pdf
Leonhard Helminger, Michael Bernasconi, Abdelaziiz Djelouah, Markus Gross, and Christopher Schroers. Generic image restoration with flow based priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognilion, pp. 334-344-44 2021.
Shady Abu Hussein, Tom Tirer, and Raja Giryes. Imagc-adaptive gan based reconstruction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3121-3129, 2020.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jonathan I Tamir. Robust compressed sensing mri with deep generative priors. arXiv preprinl or arXiv:2108.0138 2021a.
Ajil Jalal, Sushrut Karmalkar, Alexandros Dimakis, and Eric Price. Instance-optimal compressed sensing via posterior sampling. arXiv preprint arXim:2166 11.13188,, 2021b.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 8110 8119, 2020.
Varun A Kelkar, Sayantan Bhadra, and Mark A Anastasio. Compressible latent-space invertible networks for generative modellconstrained image reconstruction. JEEE Transaclions on Compulational Imaging, 7: 209 200-223,220 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htttp://arxwv..rg/aababaa Baargg/bs/14415 6980
Jianwei Ma and Francois-Xave Le Dimet. Deblurring from highly incomplete measurements for remote sensing. TEEE transactions on geoscience and remote sensing, 47 802, 2009.
Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas Vasanawala, Greg Zaharchuk, Lei Xing, and John M Panly. Deep generative adversarial neural networks for compressive sensing mri. IEEE transactions medical imaging, 38(1):167 179, 2018.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/eo) conference on computer vision and pattern recognition, pp. 2437 2445, 2020.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3481-3490. PMLR, 10-15 Jul 2018. URL hattps://proceedingg mlr. www.:preesssvve. httpa//////0mmmceeeee1BB.... html
Lukas Mosser, Olivier Dubrule, and Martin Blunt. Stochastic seismic waveform inversion using generative adversarial networks geological prior. Mathematical Geosciences, 52(():53 79, 2020.
Xingang Pan, Bo Dai, Ziwei Liu, Chen Change Loy, and Ping Luo. Do 2d {gan]s know 3d shape? unsupervised 3d shape reconstruction from 2d image {gan]s Js. In International Conference on Learning Representations, 2021. URL httpp:///peerrevie aet//Eorum??d-FFFiiDDBUXXX
Zaccharie Ramzi, Benjamin Rémy, Francois Lanusse, Jean-Luc Starck, and Philippe Ciuciu. Denoising score-matching for uncertainty quantification in inverse problems. arXiv preprint arXiv:2011.08696, 2020.
Fahad Shamshad, Asif Hanif, and Ali Ahmed. Subsampled fourier ptychography via pretrained invertible and untrained network priors. 2019.
Niklas Smedenark-Margulics Jung Yoon Park, Max Daniels, Rosc Yu, Willem van de Meent, and Paul Hand. Generator surgery for compressed sensing. ar preprinl ar 11102.111133 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. orXiv preprint
2019.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Yang Song and Stefano Ermon. Improved techniques for training score- based generative models. arXiv preprinl Xrv:22000..00 09011, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations. arXiv preprint arX.v:2011.13450, 2020.
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with a flow-based noise model. In International Conference on Machine Learning, pp. 11146-11157. PMLR, 2021a.
Jay Whang, Erik Lindgren, and Alex Dimakis. Composing normalizing flows for inverse problems. In International Conference on Machine Learning, pp- 11158-11166 PMLR, 2021b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
A Proof
Proof of Lemma 1
Notice that for any Wi E graxmi
Bxo.2 W (Wiz) 20 + =1) W, W zo | = Ezo (W Wi) 20 + Ez, |Il W 21
= [[ W Wi) + W
where the first equality used independence of zg and z1 and the second used the fact that E2~A((o | | for any matrix M E mg The solutions of equation i are then solutions of
min W WI
[[C(W)) L(Wi)
(10)
where L: R Roxxno is the linear operator given by L: W1 - WiW; Since a1 no, this operator is singular and there are infinite solutions of equation I0 all of which satisfy ((WT) L(W1), ie. the thesis.
Regarding 7 observe instead that for any W1 E R
E Zo WT (Wo + 21) Wi (W + 21) 21)1 ([ - Wi) + |WT - W.
This shows W W is the unique minimizer of
Before analyzing the solution of compressed sensing with the trained generative models, we observe the following fact the minimum norm solution of
Lemma A.1. Let Von be the minimum Frobenius norm solution of equation Then W. Van W;W, (Wi)t where (t is the pseudoinverse of Wi.
Proof. Notice that L(WT)= = L(W)Y"), where C is defined in the proof of Lemma equation We next show that W. Van orthogonal to the null space of L, which implies the thesis.
Let W1 Raa be such that L(W1) W,W = 0. Then have
(W) Van Wi) Vru(W:VanyT'
= -WWW -t<W:::((:)) =0
where third equality uses the fact that W (W;)) symmetric and the fourth equality uses the assumption W.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Proof of Lemma
Proof of equation Notice that by the previous lemma = WW WīPw; where Pw; is the orthogonal projector onto the range of W Moreover, with probability 1, (AW:Wā) is full rank. It follows that (z) Yan x van) is given by
Van arg min y AW1 Wi =612.
20
Van = arg min ly AW V)) + Pw; x
In particular Pw; 2 Van and (In, Pw; )z Van is fixed at initialization. We have then (Van 2) Van)
W + MoM where Mi = (AWi AWW
The reconstruction error then given by
Ex 2 | z1) Van (z) 22
Van Van ) [ W.W,
min
| - WWW
||( Pw; Will
where the first equality follows from the properties of the normal distribution. Regarding then the minimization problem min Mineoon [ WiWW notice that this is convex and the critical points satisfy
(W(W)) Wi (WiWG) (WT))))
Using the fact that W is full rank, the unique solution is found to be [W, (W; "W,",,,) - Wi, WW, which gives the last equality.
Note now that Pw;wy wy is the projector onto the range of W Wi and Wi full rank. Thus
[I(Ins P:;w;) Wil2 [I( P:; Will
[[ Pw;w;) 0. max kErange(W;; W
Proof of equation
Notice again that with probability 1, AW We and AW1 have full rank. Moreover Wi (WT IWr x1 ZRTII)) where
GRTH (xRTI. RTILL
20
RTIL
arg min |yy AW W =1
* =,ER0
RTIL
arg min Iyy AW RTIL (W; RTIL + +1)11
ER
It is then easy see that
RTIL 2 + (AW; AW 20
RTIL
= (AW Wi AW
where (AW:W;)) denotes the pseudoinverse of (A,:WW,) then follows that WWWW+ RTILL + 2 RTILL.
= 0
WWW 2 zī) = xx, which implies the thesis.
16

---

Published in Transactions on Machine Learning Research (01/2023)
A
Appendix
Code provided in supplementary folder. Computational requirements for this paper are two NVIDIA 2080 Ti GPU: training StyleGAN2 uses both GPU's and one GPU for inversion, training PGGAN requires one GPU and one for inversion.
A.1 RTIL-ILO Training Details
All experiments for ILO inversion method used StyleGAN2 architecture (Karras et al., 2020), both vanilla and RTIL models were trained for 500,000 itcrations using the same training parameters, i.c., learning rate, batch size, and regularization updates. Below table [ outlines the macroscopic view the StlyeGAN2 architecture. Please refer StlyeGAN2 paper (Karras et al., (020) for more architecture details, between each convolutional layer there is normalization operation called weight demodulation. During training for RTIL the distribution was induced after each block in the network, which corresponds to cells 2-6 in table I up to 4-th convolutional block.
All experiments for ILO inversion method used StyleGAN2 architecture (Karras et al. 2020), both vanilla and RTIL models were trained for 500,000 iterations using the same training parameters, i.e., learning rale, batch size, and regularization updates. Please refer to the code for more details on the training process and the StlyeGAN2 paper (Karras al. 2020) for more details on the architecture.
Below Table outliness the macroscopic view of the StlyeGAN2 architecture, between each convolutional layer there is normalization operation called weight demodulation. During training for RTIL the additional latent variables were added after each block in the network, which correspond to cells 2-6 in Table up to 4-th convolutional block.
Table 1: StyleGan2 for image size 256 x 256
Generator
Operation
Activation
Output Shape
Latent Vector
None
512x 1 x1
8x MLP (Mapping Network)
LRelu
512 x 14
Constant input
None
512 x 4 x
Conv 3 x
LRelu
256 x
Upsample
None
256 x 8 x
Conv
LRelu
256 x
Conv 3
LRelu
256 8 x
Upsample
None
256 16 16
Conv 3 3
LRelu
256 16 16
Conv 3 x 3
LRelu
256 16
16
Upsample
None
256 x 32 32
Conv 3 x 3
LRelu
256 x 32 32
Conv 3 x 3
LRelu
256 x 32 x 32
Upsample
None
256 x 64 64
Conv 3 x
LRelu
256 x 64 64
Conv 3
LRclu
256 64 64
Upsample
None
128 128 x 128
Conv 3
LRelu
128 128 128
Conv 3 3
LRelu
128 128 x 128
Upsample
None
128 x 256 x 256
Conv 3 x 3
LRelu
64 x 256 x 256
Conv 3 x 3
LRelu
64 x 256 x 256
Conv 1 x
Linear
3 x 256 x 256
Trainable Parameters 12,300,877
17

---

Published in Transactions on Machine Learning Research (01/2023)
Vanilla
G
G
GB
G
GA
Figure 8: Samples from each of our trained generative models. Top row consist samples from vanilla trained StyleGAN2, then rows GD- ... G1 are samples from family of generative models trained with RTIL.
18

---

Published in Transactions on Machine Learning Research (01/2023)
A.2 RTIL-mGANprior Training Details
All experiments for mGANprior inversion method used PGGAN Architecture (Karras et al. 2018 both vanilla and RTIL models were trained for 600,000 iterations and use the same training parameters, ie., learning rate, batch size, and regularization updates. If interested, please to refer to the code for more details. Below, table outlines the details the PGGAN architecture. The intermediate latent variable were added after the 4-th block, which corresponds the 4-th cell in Table
Table 2: PGGAN for image size 256 256
Generator
Operation
Activation
Output Shape
Latent Vector
None
512x 1 x1
Conv 4 x 4
LRelu
256 x 4 x 4
Conv 3 x 3
LRelu
256 x 4 x 4
Upsample
None
256 x 8 x
Conv 3 x
LRelu
256 x 8 x
Conv 3
LRelu
256 8 x
Upsample
None
256 16 16
Conv
LRclu
256 16
16
Conv 3
LRelu
256 16
16
Upsample
None
256 32
32
Conv 3
LRelu
256 32 32
Conv 3 3
LRelu
256 32 32
Upsample
None
256 x 64 64
Conv 3 x 3
LRelu
128 x 64 X 64
Conv 3 x 3
LRelu
128 x 64 x 64
Upsample
None
128 x 128 x 128
Conv 3
LRelu
64 x 128 x 128
Conv 3
LRelu
64 128 128
Upsample
None
64 x 256 256
Conv 3
LRelu
64 256 256
Conv 3 3
LRelu
64 x 256 256
Conv 1 x
Linear
3 x 256 x 256
Trainable Parameters: 7,445,443
19

---

Published in Transactions on Machine Learning Research (01/2023)
Vanilla
GO
G
G
Figure 10: Samples from each of our trained generative models. Top row consist samples from vanilla trained PGGan, then rows Go G3 are samples from family of generative models trained with RTIL.
A.3 Inversion Details for ILO
Hyper-parameters were tuned based on experiments the appendix of the II.O paper (Daras al. 2021 and the official GitHub repository. Our implementation uses the code from the GitHub repository, please refer to supplementary material to see. For more details about hyper-parameter choice, learning rate and number of iterations per layer were tuned for compressed sensing, then the same configuration were used for inpainting and super-resolution. For experiments in Scction 4 11 Figure g the configuration for the number of iterations per layer is {2000, 1000, 1000, 1000, 2000}. ILO-RTIL uses a learning rate that. begins al .2 al the initial layer, then ramps up linearly and ramps down using a cosine scheduler, as proposed by (Karras et al,, 2020). As for ILO, each layer initialized with a learning rate of .I then optimized independently using the same learning rate scheduler, which proposed in official Github repository (Daras al. 2021 We choose the intermediate layer up to which optimize to, based ablation study Section .3. Figure
Below
20

---

Published in Transactions on Machine Learning Research (01/2023)
report the loss function used for each inverse problem in case of ILO-RTIL and RTIL proposed in Daras et al. (2021):
Compressed Sensing Mean square
Inpainting- Equal weighted combination of mean square error and LPIPS (Zhang al. 2018 for sufficient number of measurements. Notice that with more than 50% missing pixels LPIPS did not help reconstruction performance.
Super-Resolution Equal weighted combination of mean square error and LPIPS for ILO-RTIL, for vanilla, II.O LPIPS loss weighted more with X 1.5 Except at downsampling ratio of I6 only MSE loss is utilized.
Compressed Sensing with Circulant Matrices
Inpainting
Super-Resolution
0.95
0.95
0.95
0.90
0.90
0.85
SSIM
SSIM
0.80
0.75
0.75
0.75
0.70
0.70
0.65
15
20
25
20
30
50
0.1
0.2
0.3
0.4
0.5
Percentage of Measurerments
Percentage of Observed Pixels
Downsample Factor
RTIL-ILO
ILO
Figure 11: Performance of ILO-RTIL and vanilla trained ILO for Compressed sensing, inpainting, and super resolution for various under-sampling ratios. ILO-RTIL inereases performances or tics for each under-sampling ratio with respect to SSTM across each of the inverse problems compared to IT.O. The vertical bars indicate 95% confidence intervals.
Compressed Sensing with Circulant Matrices
Inpainting
Super-Resolution
0.45
0.5
0.40
0.4
0.3
LPIPS
0.2
0.2
0.1
10
15
20
25
10
40
50
0.1
0.3
0.4
Percentage of Measurements
Percentage of Observed Pixels
Downsample Factor
RTIL-ILO
ILO
Figure 12: Performance of ILO-RTIL and vanilla trained ILO for Compressed sensing, inpainting, and super resolution for various under-sampling ratios. IT.O-RTI. increases performances or ties for each under-sampling ratio with respect to LPIPS across each of the inverse problems compared ILO. The vertical bars indicate 95% confidence intervals.
21

---

Published in Transactions on Machine Learning Research (01/2023)
A.4 Inversion Details for mGANprior
Hyper-parameters were luned based on the Github repository (Gu et al., 2020) for Section .2 Figure 15
learning rate and the number of iterations were tuned for compressed sensing then reused for inpainting and super-rosolution. mGANprior-RRIL. uses Adam (Kingma & Ba 2015) optimizer initialized at learning rate of .1 and optimized for 2500 iterations. iGANprrir uses SGD initialized at learning rate and optimized for 2500 iterations, which based on the official GitHub repository (Gu et al. 2020). Empirically the mGANprior (Gu) 20 latent codes. Moreover, for selecting
et al. 2020) improvement in reconstruction saturated after which intermediate layer to optimize over for the vanilla model was determined by the ablation study in Figure 22 Below report the loss function used for each inverse problem in case of mGGANpriorRRTH and mGANprior
Compressed Sensing Mean square error loss for both methods.
Inpainting- Mean square error plus I LPIPS regularization proposed by (Gu et al. 2020 For mGANprior-RTTI. the regularization term was X = .1 and for mGANprior .5.
Super-Resolution Mean square plus 4 LPIPS regularization proposed by GGu et al.
2020
-GAN the regularization term was scaled .1 and for mGANprior À = .5. Except at downsampling ratio of 16 where the we only use MSE loss.
mGANprior
RTIL for mGANprior
P
go
91
a
90
91
Latent variables
P=
Optimized variables
a
Entrywise addition
Channel-wise multipiictioon
P2
90
91
Parameter sharing
Figure 13: The left side portrays vanilla training for mGANprior, and the right side demonstrates RTIL. For mGANprior-RTTT. this example with N 2 latent codes, where the top model trains a vanilla model analogous to model on the left, then parameter shares with the model below.
22

---

Published in Transactions on Machine Learning Research (01/2023)
Compressed Sensing with Circulant Matrices
Inpainting
Super-Resolution
0.90
0.90
0.90
0.88
0.85
0.85
0.86
S5IM
SSIM
0.80
0.84
0.80
0.75
0.75
0.80
0.70
0.78
0.70
0.76
20
40
0
20
30
50
0.1
0.2
0.3
0.4
0.5
Percentage Measurements
Percentage of Observed Pixels
Downsample Factor
RTIL-mGANprior
mGANprior
Figure 14: Performance of mGANprior-RTTI. and vanilla trained mGANprior for compressod sensing, inpainting, and super resolution for various under-sampling ratios. mGANNrrio-RTTI. increases performances vanilla, mGANprior with respect to SSTM over each under-sampling ratio, except. for super-resolution problems at low under-sampling ratio's. s. The vertical bars indicate 95% intervals.
Compressed Sensing with Circulant Matrices
Inpainting
Super Resolutesoottion
0.45
0.50
0.50
0.45
0.45
0.40
0.40
0.35
LPIPS
LPIPS
LPIPS
0.35
0.30
0.30
0.30
0.25
0.25
0.25
0.20
0.20
0.20
0.15
20
40
60
20
30
50
0.1
0.2
0.3
0.4
Percentage of Measurements
Percentage of Observed Pixels
Downsample Factor
RTIL-ILO
ILO
Figure 15: Performance of mGANprior-RTII. and vanilla trained mGANprior for compressed sensing, inpainting, and super resolution for various under-sampling ratios. mGANppior-RTTI, increases performances over vanilla mGANprior with respect to SSIM over each under-sampling ratio, except for super-resolution problems at low inde-ssampling ratio's. The vertical bars indicate 95% intervals.
A.5 Inverse Problems Qualitative Results
23

---

Published in Transactions on Machine Learning Research (01/2023)
Ground Truth
Observations
Ours
ILO
CIO
es
SSIM=.026
SSIM=,86
SSIM=.775
SSIM=.032
SSIM=.861
SSIM=,811
Figure 16: Qualitative comparison between our method ILO-RTIL and ILO for inpainting at 5% of observod pixels. Ground Truth
Observations
Ours
mGANprior
SSIM=.055
SSIM=
SSIM=.854
SSIM=.068
SSIM=
SSIM=.894
Qualitative comparison between our method mGANprior- RTIL and mGANprior for inpainting 10% of observed pixels.
Figure 17:
24

---

Published in Transactions on Machine Learning Research (01/2023)
Ground Truth
Observations
ILO
Figure 18: Comparison between ILO-RTIL to II.O for super-resolutio LR 4x (Downsampling factor ) Ground Truth
Observations
Ours
mGANprior
Figure 19:
Comparison between mGANprior-RTIIL, mGANNrror for super-Resolution LR 4x (Downsampling
factor 1)
25

---

Published in Transactions on Machine Learning Research (01/2023)
A.6
Ablation
A.6.1
ILO-Ablation
This section corresponds to Section .3.1 Figure
the configuration for each optimization setting go as: 20 {2000), 21 {2000, 2000), 22 {2000, 1000, 2000)
23 {2000, 1000, 1000, 2000),
24 {2000, 1000, 1000, 1000, 2000} iterations per layer.
Figure 20 demonsttrats how reconstruction performance can vary based on the number of iterations per interme- diate layer while using ILO algorithm. The configuration for EXP 1-5 goes as following in order of number of it- crations per intermeidate layer: EXP {100, 100, 100, 100, 100), EXP = {300, 300, 300, 300, 300), EXP 3 { [1000, 1000, 1000, 1000, 1000), EXP {2000, 1000, 1000, 1000, 2000 EXP = {2000, 2000, 2000, 2000, 2000}.
Compressed Sensing with Circulant Matrices for Various Intermediate Layer Iterations
RTIL EXP
Van EXP
38
Van EXP 2
RTIL EXP
Van EXP 3
Van EXP 4
34
RTIL EXP
Van EXP
PSNR
30
26
22
10
Percentage of Measurements
Figure 20: Compressed Sesning with Circulant Matrices reconstruction performance for various number of iteration per intermediate layer with ILO.
26

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 21 demonstrates how reconstruction performance can be affected based on the size of { balls used in II.O algorithm [ The number of iterations for each of the experiments {2000, 1000, 1000, 1000, 2000) corresponding to each intermediate layer, which The same used in the paper. Size of the & balls were determined based on the author's rocommendation in the appendix [Daras et al. (2021) p. EXP 1-4 goes as following order of number intermeidate layer and the corresponding radius of the size of search space for the & ball of radius of noise, latent code, and projection of the 1-{100, 200, 500, 1000, 2000);(100, 200, 300, 800, 1000),{100; 200, 300, 800, 10003.2- {200, 1000, 100, 2000, 4000),(200, 400, 800, 1600, 3200), 400, 800, 1600, 3200).3- (1500, 1500, 2500, 4000, 5000),(3000 500, 1000, 2000, 3,00), 1200, 2000, 3000, 5000),4
previous solution:
{500, 1500, 2500, 4000, 5000), 750, 1500, 2500, 4000), (500,, 750, 1500, 2500, 5000}.
Compressed Sensing with Circulant Matrices for Various 4 Constraints
38
36
34
32
PSNR
30
RTIL No Constraints
26
Vanilla No Constraints
EXP 1
24
EXP 2
EXP 3
EXP 4
5
15 Percentage of Measurements
10
Figure 21: Compressed Sesning with Circulant Matrices reconstruction performance for various & constraints.
27

---

Published in Transactions on Machine Learning Research (01/2023)
A.6.2
mGANprior-Ablass -
Inpainting
Suplr-Resolution
27
27
26
26
25
PSNR (dB)
PSNR (dB)
25
24
23
24
22
23
21
22
20
10
20
30
40
50
0.1
0.2
0.3
0.4
0.5
Percentage of Measurements
Downsampling Factor
mGANprior Layer mGANprior Layer 4
mGANprior Layer mGANprior Layer
Figure 22: Effects in-painting and super-resolution performance for various intermediate layer on validation set of 5 images.
28


Reviews:
[{"review_id": "74Mh4xBr0A", "reviewer": "Reviewer_RRRQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The idea of the paper is novel and intuitive. Prior work has shown that i) extending the generator range and ii) regularizing the solutions are both important steps for solving inverse problems. This paper proposes a regularization during the training phase.\n* The proposed approach maintains the flexibility of solving inverse problems with deep priors; the network is only trained once and then can be used to solve many inverse problems.\n* The paper is clearly written.\n* The comparison with the baselines is thorough. The authors compare in many tasks, including Compressed Sensing, Inpainting and Super-Resolution. The authors run a fair amount of ablation studies to show that the comparison with the baselines is fair.\n\n\nWeaknesses:\n\n* There is an \"unfair\" advantage of this work over ILO and multi-code prior, namely the authors of this work are allowed to change the generator during training. It seems reasonable that by doing so the authors are able to obtain better results for inverse problems.\n* To use this approach, one needs to change the training of the model. That can be problematic for a couple of reasons. First, it is quite common that the weights of Foundation Models are released without their training data. In that case, the loss of Eq. (2) cannot be used. Second, training a model (or even finetuning it) is much more computationally expensive compared to solving an inverse problem. \n* Since the networks used in the paper are trained from scratch, it would be good to see their unconditional performance, e.g. maybe report FID scores. Sometimes, improved performance in inverse problems comes purely from having access to better generators.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel way of solving inverse problems with GANs. The paper builds upon two well-established prior works, ILO and Multi-Code Prior. Both prior works attempt to increase the expressive power of the generator by optimizing intermediate layers. The authors of this work argue that when doing so, the intermediate layers need to be regularized during training. The propose a novel adversarial loss that feeds to the discriminator samples from both the vanilla model and the \"expanded\" model with intermediate latents coming from a simple distribution. This imposes constraints on the intermediate learned manifolds and experimentally yields performance benefits over the baseline methods. Theoretically, the authors analyze a toy setting under which they prove that their method is provably superior to ILO.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* From Figure 1, it seems that first the network is trained with vanilla training and then finetuned. In the experimental section, it seems that the regularized models are trained from scratch (?). Could the authors please clarify? How is the performance impacted when the regularized models are trained from scratch vs. being finetuned from a pre-trained checkpoint?\n\n* For the models trained from scratch, I would like to see unconditional FID scores. The objective here is not to achieve state-of-the-art, but to eliminate the possibility that better performance in inverse problems comes from having a better unconditional generator. \n\n* The proposed regularization is to sample intermediate latents from a simple distribution during training. However, during inference, the optimization in the intermediate latent space is unconstrained (as far as I understand). It would be very interesting to see if the latents obtained by solving inverse problems are actually closer to being sampled from a Gaussian in the case of RTIL-ILO vs ILO.\n\n* For the toy setting proof, how would things change if $W_0^*$ was also an optimization variable? It seems that a central argument for the proof relies on the fact that the RTIL-ILO has a unique solution (for this toy setting), while the vanilla method has infinite solutions that might introduce errors in certain directions. I would like the authors to clarify this point.\n\n* From the plots of Figure 4, it seems that RTIL-ILO has a bigger advantage over ILO when the measurements are high. What is the intuition behind this observation? I would think that regularization is less required as more and more measurements are observed. Could the authors give numbers on what happens when even more measurements are available? Is ILO finally catching up to RTIL-ILO?\n\n* Minor: there are some typos in Equations (6), (7). ", "broader_impact_concerns": "I do not have any concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bd55Hr6VoG", "reviewer": "Reviewer_UPvB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n* the proposed approach is intuitive and is generic to an array of signal inversion / reconstruction methods\n* Empirical results on multiple existing image inversion methods look promising\n\nWeakness\n* The technical contribution is limited, as the proposed method is a straightforward extension of existing methods. \n* The evaluation is very limited from multiple aspects, including\n  * Models: the experiments only consider two models, one for each inversion method\n  * Data and task: the experiments are performed only on FFHQ and on a small number of related tasks\n  * Test data size: it is very small and not comparable with related works in super resolution, etc.\n  * Evaluation metrics: only PSNR is reported, while other standard metrics like LPIPS, FID, are not considered\n  * Baselines: the proposed method is not compared with other existing methods for the inversion problem and methods for inpainting / super-resolution / compressed sensing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how to improve a specific class of signal inversion methods based on generative models. More specifically, it improves the inversion methods that reconstruct the original signal by optimizing the intermediate features of a generative model. \n\nWhile generative models are trained to generate a target output $y$ from an input $x_0$, i.e. $y = g(x_0)$, where $g$ is a composition of multiple functions $g_i(*)$: $g(x_0) = g_t \\circ g_{t-1} \\circ ... \\circ g_1 \\circ g_0(x_0)$, these methods recover not only $x_{0}$ but also $x_{i}$ in order to recover the true $y$, i.e. from \n\\begin{equation}\narg min_{x_0} || y - g(x_0) ||\n\\end{equation}\nto\n\\begin{equation}\narg min_{x_0, ..., x_{i}} || y - g(x_0, ..., x_{i}) ||.\n\\end{equation}\n\nHere $x_{i}$ stands for the input of $g_i(\\*)$. This can be achieved by relaxing $x_{i} = g_{i-1} (x_0)$, e.g. $x_{i} = g_{i-1} (x_0) + z_{i-1}$, where $z_{i-1}$ are additional variables in the signal reconstruction stage and are not used during training $g(*)$.\n\nThe main idea of the proposed method is, instead of training $g(x_0)$, we should train $g(x_0, ..., x_{i})$ directly for the inversion problem. Empirical results on two existing methods show that the proposed method outperforms the baselines using $g(x_0)$.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given the limited technical contribution, I think a more detailed empirical evaluation is necessary for acceptance. In particular, the authors should:\n* Evaluate the proposed method on more diverse tasks and data. Face images are not sufficient to demonstrate the usefulness of the proposed method.\n* Evaluate on larger datasets with standard metrics to really verify the performance.\n* Compare with state-of-the-art methods for the tasks used in evaluation.\n\nAlso, I think it is not sufficient to show that the proposed method improves the corresponding baseline. The authors should explain why this is important and the improvement really matters in practice. ", "broader_impact_concerns": "No specific concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BEEIqlEV8Y", "reviewer": "Reviewer_dJAR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is clearly written, with nice figure illustration. The idea is a natural but meaningful extension over existing GAN inversion methods, and it shows some promising results on different tasks. \n\nWeakness:\n\n1. Not discussing the details of computational overhead. I expect that the inversion results will be better when we introduce the new training method, however, training a StyleGAN can be very expensive. In contrast, previous inversion methods can use pre-trained models. Therefore, it is important to discuss this trade-off. How much computational overhead is introduced?\n\n2. Related to previous one. It is questionable that whether it worths to train a new model, given that the improvements are not very significant. In particular, multiple previous work train generative models directly to solve inverse problems (instead of serving as a prior) by modeling the conditional distribution. Since your method now requires training a new generative model, those method should be compared with. For example, super-resolution [1] and inpainting [2]. These are based on diffusion models, but there are many others based on GAN. \n\n3. Evaluation metric. Work on using generative models to do super-resolution or inpainting need to report FID, rather than single-image metric like PSNR on a few images. It is insufficient to evaluate on a few images as that may result from cherry picking.\n\n[1] Image Super-Resolution via Iterative Refinement https://arxiv.org/abs/2104.07636\n\n[2] RePaint: Inpainting using Denoising Diffusion Probabilistic Models, https://arxiv.org/abs/2201.09865", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose to train the intermediate layers of GANs in order to do better GAN inversion, which can be further applied to inverse problems by serving as a generative prior. The method is based on existing GAN inversion methods such as Intermediate Layer Optimization and Multi-Code Prior. These methods optimize over intermediate layers by introducing an additional optimization variable, and  RTIL further trains the GAN model with GAN objective. In other words, it optimize the GAN parameters with sampling from the extra latent variables. Results show some improvements in GAN inversion (reconstruction error) and applications to inverse problems (super-resolution, in-painting, compress sensing).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clearly discuss the computational overhead issue.\n\n2. Compare with generative models that solve inverse problem directly, and with more meaningful metric such as FID.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission looks into the problem of model inversion via optimization of intermediate activation values in generative adversarial networks (GANs). It proposes to make GANs more suitable for this kind of model inversion through regularized training (a technique called Regularized Training of Intermediate Layers, or RTIL). RTIL works by incorporating the latent variables introduced by inversion techniques like ILO or mGANprior into the training procedure itself. The GAN loss term which operates on generated data is replaced with a convex combination of loss terms which operate on data produced by the unmodified generator and data produced by the generator augmented with the additional latent variables prescribed by the chosen inversion algorithm.

The approach is evaluated and compared against the baseline of no regularized training for two pairs of inversion algorithm and generator architecture (ILO and StyleGAN2, mGANprior and PGGAN) and for three tasks (compressed sensing, inpainting, and super-resolution). Ablations are run to measure the efficacy of the proposed approach in comparison to the baseline in terms of number of latent codes introduced. Finally, theoretical results are presented in a toy setting to justify the soundness of the proposed approach.

Reviewers note the presentation's clarity (dJAR, RRRQ), the proposed approach's broad applicability (UPvB) and its flexibility (RRRQ), and the promising (UPvB) and thorough (RRRQ) empirical results.

Concerns raised by reviewers include:

- Reviewer RRRQ raises questions on the RTIL workflow shown in Figure 1, which the authors address in their response.
- Reviewer RRRQ asks for FID metrics for unconditional generation in order to rule out the possibility that the improvements observed simply stem from a better generator. The authors provide those metrics in their response.
- Reviewer RRRQ notes that the approach is not applicable to foundation models for which the training data is unavailable. While this is true, this does not detract from the paper's main claims.
- Reviewer UPvB finds the submission's technical contribution somewhat limited. TMLR's acceptance criteria are a) claims supported by accurate, convincing and clear evidence and b) interest to at least some individuals in TMLR's audience, therefore this concern is beyond the scope of TMLR's criteria.
- Reviewer UPvB is concerned that the evaluation is too limited in terms of models, tasks, data regimes, metrics, and baselines. The authors respond by increasing the test set size for CELEBA-HQ and providing LPIPS metrics. This does not satisfy Reviewer UPvB, who remains concerned that the empirical evidence is insufficient.
- Reviewer dJAR asks for a discussion of the computational overhead associated with the proposed approach, which the authors provide in their response.
- Reviewer dJAR is concerned that the proposed approach is not compared against approaches which train generative models directly to solve the inverse problem. The authors respond that while interesting, such a comparison is beyond the scope of the claims made in the paper.

Ultimately, all reviewers indicate that the submission is of sufficient interest to at least some individuals in TMLR's audience, and most reviewers indicate that the submission's claims are supported by accurate, convincing, and clear evidence. While I agree with Reviewer UPvB that expanding the evaluation would strengthen the submission, I side with the majority opinion that the empirical evidence presented is sufficient to support the paper's main claims.
Decision: Accept (poster)