Optimizing Intermediate Representations of Generative Models for Phase Retrieval
Authors: Tobias Uelwer
Sebastian Konietzny
Stefan Harmeling
Venue: Accepted by TMLR ()

Abstract: Phase retrieval is the problem of reconstructing images from magnitude-only measurements. In many real-world applications the problem is underdetermined. When training data is available, generative models allow optimization in a lower-dimensional latent space, hereby constraining the solution set to those images that can be synthesized by the generative model. However, not all possible solutions are within the range of the generator. Instead, they are represented with some error. To reduce this representation error in the context of phase retrieval, we first leverage a novel variation of intermediate layer optimization (ILO) to extend the range of the generator while still producing images consistent with the training data. Second, we introduce new initialization schemes that further improve the quality of the reconstruction. With extensive experiments on the Fourier phase retrieval problem and thorough ablation studies, we can show the benefits of our modified ILO and the new initialization schemes. Additionally, we analyze the performance of our approach on the Gaussian phase retrieval problem.

Published in Transactions on Machine Learning Research (11/2022)
Optimizing Intermediate Representations of Generative Models for Phase Retrieval
Tobias Uelwer*
abkias.weverrttt-ddrtmwdddd
Deportment of Computer Science Technical University of Dortmund
Scbastian Konictzny"
Department Computer Science Technicol University of Dortmand
Stefan Harmeling
Department of Computer Science Technical University of Dortmand
Reviewed on OpenReview: /ttps://opents optp:/veew..eeeee.eoeoe net/ forum? YAVE6jfeJb
Abstract
Phase retrieval is the problem of reconstructing images from magnitude-only measurements. In many real-world applications the problem is underdetermined. When training data is available, generative models allow optimization in a lower-dimensional latent space, hereby constraining the solution set to those images that can be synthesized by the generative model. However, not all possible solutions are within the range of the generator. Instead, they are represented with some error. To reduce this representation error in the context of phase retrieval, first leverage novel variation of intermediate layer optimization (ILO) to extend the range of the generator while still producing images consistent with the training data. Second, we introduce new initialization schemes that further improve the quality of the reconstruction. With extensive experiments on the Fourier phase retrieval problem and thorough ablation studies, we can show the benefits of our modified ILO and the new initialization schemes. Additionally, we analyze the performance of our approach on the Gaussian phase retrieval problem.
1 Introduction and Related Work
Optimizing intermediate layers in generative models can lead excellent results in various linear inverse problems like inpainting, super-resolution, denoising and compressed sensing, as shown by Daras et al. [ Tn our paper, we show that their approach be extended to solve the more difficult non-linear inverse problem of phase retrieval. In particular, we also lackle the challenging problem of non-oversampled phase retrieval, which we explain next.
For simplicity, we define different instances the phase retrioval problem for the onc-dimensional case. However, extensions to Iwo or more dimensions are straight-forward.
Fourier Phase Retricval: In many imaging applications in physics, we are only able to mcasure the magnitude of the Fourier transform of an image, e.g., in astronomy 10,, X-ray erystallography 25
optics 35 In the Fourier setting the magnitude measurements y R are given as
y Arl
(1)
"equal contribution

---

Published in Transactions on Machine Learning Research (11/2022)
where A= F E C=x" is the discrete Fourier transform (DFT) matrix and R" is the original image (here represented as a vector). However, due to the symmetry of the DFT, has only Ln/2] + many distinct values and thus recovering the entries of æ is underdetermined. Thus we require additional information about z, which, in this paper, will be learned from example images to constrain the solution space.
In contrast, the common assumption in Fourier phase retrieval that x zero-padded and thus the problem is no longer underdetermined. The zero-padding of leads an oversampling of the relevant signal and makes the problem much easier to solve. Again, in this work, assume that the Fourier measurements are not oversampled, Le., consider the case, where the image I has not been zero- pero-padded and entries need to be recovered.
Compressive Gaussian Phase Retrieval: A related phase retrieval problem is the compressive Gaussian phase retrieval problem (discussed in the work of Candes et al. | and Shechtman et al. (0]. ). For this problem, the DFT matrix is replacod by matrix A with randomly sampled entries (real- or complex-valued, typically sampled from a Gaussian distribution), i... the measurements y R are given by
y y= |Arl
(2)
If the number of entries of does not exceed the number of entries of x, ie., A E R has fewer rows than columns, the problem is said to be undersampled.
Note, that both variants of phase retricval are more difficult than linear inverse problems due to the non- linearity (the absolute value) in the forward model. Both problems have received lot of attention in the literature. In the following, we give short overview of the various approaches distinguishing between classical oplinizaaion--assed and learning-basee based methods.
1.1 Methods without Learning for Phase Retrieval
One of the first (Fourier) phase retrieval algorithms is Fienup's error-reduction (ER) algorithm [+] that is based on alternating projections. In his work, Fienup [1] also introduced the hybrid-innuut-output (HTO) algorithm which improved the ER algorithm. The Gaussian phase retrieval problem was approached by Candes et al. [ who used methods based on Wirtinger derivatives to minimize a least-squares loss. Wang et al. 366 considered similar idea but used a different loss function and so-called truncated generalized gradient iterations. Holographic phase retrieval is a related problem, which aims to reconstruct images from magnitude measurements where known reference is assumed to be added onto the image. This problem was approached by Lawrence et al. 19 using untrained neural network priors. Untrained neural networks have also been used by Chen et al. 5 for solving different phase retrieval problems. Phase retrieval using the RED framework 29] was done by Metzler et al. [24], Wang et al. 37 and Chen et al. G. However, these works focus on the reconstruction of images from coded diffraction pattern measurements or 4x oversampled Fourier magnitudes.
1.2 Learning Lease Methods for Phase Retrieval
Recently, learning-based methods for solving phase retrieval problems gained lot of momentum. They can be classified into two groups: supervised methods and unsupervised methods.
Supervised methods: These methods directly learn a mapping that reconstructs images from the mea- surements. Learning neural networks for solving Fourier phase retrieval was done by Nishizaki al. pr] This approach was later extended to neural network cascade by Uelwer et al. 32 who also used conditional generative adversarial networks to solve various phase retrieval problems R While giving impressive
results, all of these supervised approaches have a drawback for the Gaussian phase retrieval setup, since the measurement operator must be known at training time. Supervised learning of reference images for holographic phase retrieval was done by Hyder et al. M and Uelwer et al. gave further insights.
Unsupervised methods: Unsupervised methods are agnostic with respect to the measurement operator, because they are trained on dataset of images without the corresponding measurements. Generative
2

---

Published in Transactions on Machine Learning Research (11/2022)
models for Gaussian phase retrieval have been analyzed by Hand et al. B3 and Liu al. 21 Killedar & Seelamantula [8] apply sparse prior on the latent space of the generative model to solve Gaussian phase retrieval. Alternating updates for Gaussian phase retrieval with generative models was proposed by Hyder et al. 14. Manckar et al. 23 used passive loss formulation to tackle the non-oversampled Fourier phase retrieval problem.
In the context of these related works, the method that we propose in this paper can be seen an unsupervised learning- based approach that can optionally be extended with a supervised component, as detail in Section 2.2
1.3 Optimizing Representations of Generators for Inverse Problems
Generative models have been used to solve linear inverse problems, e.g., compressed sensing 3. image inpainting -T. To decrease the reprosentation error and thereby boost the expressiveness of these generative models, the idea of optimizing intermediale represenlations was successfully applied in the context of compressed sensing [T,, image inpainting and super-resolution [8] The former paper called this approach generator surgery, the latter intermediate layer optimization (ILO).
1.4 Our Claims and Contributions
The claims made in this paper are the following:
We claim that the idea of optimizing intermediate representations in generative models be applicd to solve phase retricual problems. However, given the difficully of these non-linear inverse problems an additional subsequent optimization step is required to obtain good results.
Claim 1:
with random restarts for generator-bused approaches to solve inverse problems can be reduced by using different initialization schemes.
Claim 2: We also claim that the need for multiple
Further intuition and motivation will be given in Section 2.1
In extensive experiments, we consider the underdetermined Fourier and Gaussian phase retrieval (with real and complex measurement matrices) and show that our method provides state-of-the-art results. By analyzing various combinations of methods and generator networks, we show the influence of the choice of the generator. In further ablation studies, we show the importance of each component of our method.
2 Phase Retrieval with Generative Models
The basic idea of applying a trained generative model G R' + RR" to the phase retrieval problem is to plug G(z) into the least-squares data fitting term and to optimize over the latent variable 2, i.c..
min |||AG()) v/|2
(3)
This method has been used for Gaussian phase retricval (with real-valued A) by Hand al. B While this approach yields good results, the reconstruction quality is limited by the range of the generator network G. This issue is, for example, also discussed in the work of Asim et al. [2]. In the following, explain how the range of G can be extended by optimizing intermediate represcntations of G instead of only solving Equation
3 with respect to the latent variable z.
Notation: For the exposition of the method we use the following notation: The generator network G G=G o...o G1 can be written as the concatenation of layers GI...., Gt. The subnetwork consisting of layers through j is denoted by G G, o... G, Note, that = G G o G for sii k. The output of layer G, written as 2i, ie., 2 G((z-1). The input of G is 20.
Furthermore, we define the e ball with radius 0 around as,
Bx(x)
fx j 2||1 $ sr)
(4)
3

---

Published in Transactions on Machine Learning Research (11/2022)
B: Back-projection
A: Forward optimization
Gf
G
G1
G2
G3
G₁
G(2)
C: Refinement
Figure Structure of the generator network and, blue, the parts relevant for steps A, and C.
2.1 Phase Retrieval with Intermediate Layer Optimization (PRILO)
The key idea of ILO is to vary the intermediate representations learned by a sub-network of the generator G while ensuring an overall consistency of the solution. The latter is particularly challenging for non-lincar problems like phase retrieval. As we will show in the experiments, additional steps and special initialization schemes essential to obtaín useful results. All of these steps will be explained next.
The generator network captures the prior knowledge and restricts the solutions of the (underdetermined) phase retrieval problem. In this work, we use two important ideas:
Range extension: Hidden representations 2; for > 0 are allowed to vary outside the range of G This will reduce the representation error the generator.
Image consistency: Solutions must be realistic, nou-degenerate and similar to the the training dataset. This is achieved by back-projection ensuring that z, not too far away from the image of some 2p.
Note that has to strive for tradc-off between range extension and image consistency, where range extension facilitates the optimization, while image consistency regularizes the solution.
Coneretely, we repeatedly split the generator G into sub-networks Gi and G and itcratively optimize the intermediate representations resulting from G (instead of simply optimizing the initial latent variable zn). In this way, we go outside the range the sub-network G to reduce the mean-squared magnitude error. To make the resulting image more consistent we back-project 2; closer the range of Gj. We will show that zg from the back-projection not only a good candidate for the overall optimization, but also serves as good initialization for further optimization leading to significant. refinement of the overall solution.
To get started, we initially optimize the input variable (minimize Equation 3) This provides starting points for all intermediate representations for > 0. The overall procedure then iteratively applics the following three steps to different intermediate layers:
A. Forward optimization: vary the intermediate representation z; to minimize the magnitude error while staying in the ball with radius r; around the current value of 24, ie.,
arg min ||AG 2
5
=c (x.)
Note, that this step possibly leaves the range of G (range extension). By introducing ball-constraints on the optimized variable we avoid overfitting the measurements.
B. Back-projections find an intermediate representation (20) that is close to the optimal 2, from step A, i.e.,
5 = arg min |
x
(6)
=EB (0)
4

---

Published in Transactions on Machine Learning Research (11/2022)
where 0 E R'' denotes the vector of all zeros. By doing so ensure that there exists latent that yields the reconstruction. Thereby, we regularize the solution from step A to obtain image consistency.
While these two steps have been shown improve the overall performance of linear inverse problems, we found that for phase retrieval the following refinement step significantly improves the reconstructions.
C. Refinement: Our intuition here is that the back- pooc--prjected latent &y serves as a good initialization for further optimization. Starting from the Eg found in step B, we again optimize the measurement error
z arg min ||AG(=)] vll2
c (5o)
Note, that as we only optimize the latent variable of G, we do not have to apply the back-projection again for this refinement step.
These steps are applied repeatedly to the different intermediate representations. Multiple strategies for selecting the layers to optimize are possible. In practice it turned out to be sufficient to treat the layers once from left to right. Thus, we recommend to start with the layers closer to the input and then progress further with the layers closer to the output. Starting with the later layers can cause problems as they offer too much flexibility and no regularizing effect. The final reconstruction obtained as x G(zz). Steps A, B and C are visualized in Figure
The three optimization problems stated in Equations 1 solved using projected gradient descent. This means that after each gradient descent step the iterate is projected back onto the appropriate 4-ball. In our implementation this projection is implemented using the method described by Duchi et al. 9
2.2 Initialization Schemes
Due to the difficulty of the phase retrieval problem, many methods are sensitive to the initialization. Candes et al. ! proposed spectral initialization technique that is often used for compressive Gaussian phase retrieval. Since it requires one to solve a large eigenvalue problem, this method can be quite costly. A common approach to improve reconstruction performance is to use random restarts and to select the solution with the lowest magnitude error after the optimization times. Instead, we propose two fast initialization schemes that use the generative model and the available magnitude information before the optimization.
13 However, this requires running the method multiple
Magnitudde-Informed Initialization (MII): Deep generative models can generate a lot of images relatively low cost: instead of optimizing Equation with random restarts, sample set of starting points Z =(100) ....,.00) = and select the one with the lowest magnitude mean-squared error (MSE),
Jnit 20
arg min |||AG(=) - 2 2E2
(8)
before the optimization. The empirical reason for this choice is that the MSE between the unknown target image and the initial reconstructions G(z) for Z strongly correlates with the MSE of their magnitudes (correlation 0.91). This initialization scheme is applicable to other reconstruction algorithms that search in the latent space of generative model as well.
Learned Initialization (LI): The generator is trained on datasct of images that are characteristic for the problem. We use the generator to train an encoder network Eg that maps magnit measurements y to latent. representations zq- Once the encoder is trained, we can use to predict an initializtion for the optimization discussed in Section 2..1
A naive way to train the encoder network is to create input/ input/output pairs by starting with random latent vectors 2g and combining them with the magnitudes [AG(20)] of the corresponding image. The disadvantage
5

---

Published in Transactions on Machine Learning Research (11/2022)
is that the original training images only implicitly used in this approach (because those were used to train the generator). To leverage the generator and also the training images, estimate the weights 0 of the encoder Eg, such that encoded magnitudes Eg(y) generate an image G(Ea(y)) that is close to the original image z. This idea originates from GAN inversion 4
More precisely, we minimize a combination of three loss functions to train the encoder
LMSE (Goo(G(Eo()))... + (G(oC + Andv (atrr (DG(G(()()))
(9)
where LMSE is the image MSE, Lpere the LPIPS loss 12 and Ladv is a Wasserstein adversarial loss [ with gradient penalty 12 (using the discriminator network Da)- ) In our experiments, we set Apere = 5 10-5 and Aadv = 0.1. Note, that the generator network fixed and we only optimize the encoder weights and the discriminator weights ø solve a learning objective.
Additionally, also found helpful to apply small normally-diitrrbuteed pertarbation with mean 0 and standard deviation = 0.05 to the predicted latent representation. We do because using a fixed initialization leads to deterministic optimization Irajectory when rumning the optimization multiple times.
For better exploration of the optimization landscape we also apply gradient noise during optimization. In combination with the projection onto the feasible set the update reads as
(k+1) = P (2(2 10:0-aal (Vavv (2 (Vas(2!t) + vu()))
(10)
where f is the objective function corresponding to the current step, (8) (O rf with of (1+1)7 and P is the projection onto (z))))) In our experiments set 17 0.02 and 7 0.55. This noise decay schedule was proposed by Welling & Teh 39 and is also discussed by Neelakantan et al. 26 in the context of neural network training.
One drawback of the learned initialization is that requires one to retrain the encoder network when the measurement matrix changes. In the following, we only evaluate this approach for the Fourier phase retricval problem.
3 Experimental Evaluation
We evaluate our method on the following datasets: MNIST 20, EMNIST M. FMNIST [4] and CelebA [2]. The first three datasets consist of 28 28 grayscale images, whereas the latter dataset is collection of 200, 000 color images that we cropped and rescaled to resolution of 64 x 64.
Similar to Hand et al. 13 use fully-connected variational autoencoder (VAE) for the MNIST-like datasets and DCGAN 28 for the CelebA dataset. Both architectures were also used by Bora et al. 3 for compressed sensing. Details about the VAE training are described in Appendix A Going beyond existing works, we are interested in improving the performance on the CelebA dataset even further, thus we considered deeper, more expressive generators, like the Progressive GAN [G and the StyleGAN [17] Since the optimization of the initial latent space for deeper generators is more difficult, we expect significantly better results by adaptively optimizing the successive layers of these models. Our implementation is based open-source projects M In total our computations took two weeks on two NVIDIA A100 GPU. Detailed hyperparameter settings can be found in Appendix G
3.1 Phase Retrieval with Fourier Measurements
For the problem of Fourier phase retrieval, we compare our method first with the ER algorithm [|] and the HTO algorithm IT (both having learning component) and then with the following supervised learning methods: an end-to-end (E2E) learned multi--layer-
33 residual network 27, cascaded
httpp////tthiocoo/aaaaaallrrrres pytorch
httpe//gttaab..co//raaalllttt/t/ based gan gan-pytorch

---

Additionally, we apply DPR which was Published in Transactions on Machine Learning Research (11/2022)
Table 1: Fourier phase retrieval: Our proposed method (being unsupervised) performs best among classical, unsupervised and supervised approaches for MNIST and EMNIST, and second best for FMNIST (in terms of PSNR). Best values are printed bold and second-best are underlined.
MNIST
FMNIST
EMNIST
Method
Learning PSNR (t)
SSIM (t)
PSNR (t)
SSIM (t)
PSNR (t)
SSIM (t)
ER 1
15.1826
0.5504
12.7002
0.3971
13.1157
0.5056
HTO
23.4627
0.5274
12.9530
0.4019
14.2230
0.4942
DPR
unsup.
23.9871
0.9005
18.8657
0.6846
20.9448
0,8568
PRILO (ours)
unsup.
42.7584
0.9843
20.2087
0.7133
33.8263
0.9367
PRILO-MII (ours)
unsup.
43.9541
0.9949
21.4836
0.7560
37.1807
0.9719
ResNet
27
sup.
17.0886
0.7292
17.4787
0.6070
14.4627
0,5616
E2E
DN
sup.
18.5041
0.8191
20.2827
0.7400
17.4049
0.7598
CPR
32
sup. 19.6216
0.8529
20.9064
0.7768
19.3163
0.8537
PROGAN*
33
sup. 41.3767
0.9890
25.5513
0.8376
27.1948
0.9416
multilayer-perccne perceeprro 32 and a conditional GAN approach
33
originally only lested on Gaussian phase retrieval
13
Table and Table show the mean peak--signal-to-noisee roise (PSNR) and the mean structural similarity index mcasure (SSIM) 3S Results with 95%-confidence intervals can be found in Appendix E, Each reportod number was calculated on the reconstructions of 1024 test. samples. We allow four random restarts and select see, more expressive models
the generated sample resulting in the lowest measurement error. As note that our proposed initialization schemes, i.e,, the LI
result in better reconstructions. Furthermore, and MII, lead to improved performance. Figure 2 shows typical reconstructions from the CelebA dataset. Reconstructions using the DCGAN and the Progressive GAN can be found in Appendix and Appendix C
Surprisingly, our unsupervised PRILO-MII approach often quantitatively outperforms the supervised com- petitors (MNIST and EMNIST). Only for FMNIST we get slightly worse results which we attribute to the performance of the underlying generator of the VAE. Notably, we outperform DPR that uses the same generator network, which shows that the modifications explained in this paper beneficial. Summarizing, among classical and unsupervised methods our new approach performs best, often even better than the supervised methods.
The CelobA dataset is more challenging and the influence the choice the generator networks is substantial. We consider the DCGAN 28, the Progressive GAN T6 and the StyleGAN Unn as base models for our approach (second column in Table 2). Furthermore, the new initialization schemes have a strong impact. For fairness, we also combined the existing DPR approach [3] with the various generator models and the initialization schemes. Thus are able to study the influence of the different components. Among the generators, StyleGAN performs best. MIT improves the results, while LI achieves even better reconstructions. For all previously mentioned combinations, our new method PRILO is better than DPR.
To further support our claims, we performed additional experiments on the high-resolution FFHQ dataset using the StyleGAN architecture. The results can be found Appendix F] Although, the reconstruction capabilities of all considered methods are still limited, our proposed changes improve the reconstructions.
3.2 Phase Retrieval with Gaussian Measurements
Beyond Fourier phase retrieval, compressed Gaussian phase retricval is a similarly challenging problem, where the measurement matrix A has real- or complex-valued normally-distributee entries. For the real-valued case, we sample from a zero-mean Gaussian with variance 1/m,
A (ak))==I..... k= N El....
(11)
m

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2: Fourier phase retrieval on CelebA: Our proposed PRILO combined with LI and StyleGAN achieves the best results in terms of PSNR and SSIM, also against enhanced version of DPR, which uses LI and StyleGAN, Best values are printed bold and second-best are underlined.
CelebA
Method ER
Base model
Learning
PSNR (f)
SSIM (?)
10.4036
0.0637
HIO
10.4443
0.0510
DPR 13
DCGAN
unsup.
16.9384
0.4457
DPR-MIT
DCGAN
unsup.
17.8651
0.4898
PRILO
DCGAN
unsup.
18.3597
0.5159
PRILO-MII
DCGAN
unsup. 18.5656
0.5264
DPR
Progressive GAN
unsup.
18.4384
0.5276
DPR-MIT
Progressive GAN
unsup.
19.5213
0.5738
PRILO
Progressive GAN
unsup. 19.2779
0.5665
PRILO-MII
Progressive GAN
unsup. 19.9057
0.5910
DPR
StyleGAN
unsup 18.2606
0.4859
DPR-MII
StyleGAN
unsup.
20.3889
0.5972
PRILO
StyleGAN
unsup.
18.5542
0.5143
PRILO-MII
StyleGAN
unsup.
21.4223
0.6358
E2E 33
sup.
20.1266
0.6367
PROGAN
30
sup.
22.1951
0.6846
DPR-LI
StyleGAN StyleGAN
sup.
22.6223
0.7021
PRILO-LI
sup.
23.3378
0.7247
HIO 1
E2B 8
PRCGAN
35
DPR 13
PRILO-MIT (ours)
PRILO-LI (ours)
Target
Figure 2: Fourier measuroments on CclebA: only PRILO-LI (based on StyleGAN) is able to reconstruct finer details of the faces and produces realistic images with few artifacts.

---

Published in Transactions on Machine Learning Research (11/2022)
MNIST: real-valued measurement matrices
CelebA: real-valued measurement matrices
PSVR
25
20
15
10
MNIST: complex-vlued measurement matrices
CelebA: complex-valued measurement matrices
45
DPR-MII
40
PRILO
35
PSWR
15
100
200
300
400
500
500
1000
1500
2000
2500
3000
Number measurements
Number measurements
Figure 3: Real and complex Gaussian phase retrieval on MNIST and CelebA: PRILO-MIT greatly improves DPR's reconstruction results. Note, that we also combine DPR with our MII. For MNIST we used the VAE and for CelebA the StyleGAN base model.
where m the number measurements. The entries of the complex-valie valued measurement matrix are obtained by sampling real and imaginary parts from a zero-mean Gaussian with variance 1 1/(2m),
A = (ax + k= with aui, but ~ N O, 2m
(12)
1=.....
We directly compare our results with the results of the DPR method for real- and complex-valued measurement matrices (the real-valued case was already covered in [ Figure 3 shows the PSNR and the SSIM values for different nunbers of measurements m. Our proposed method PRILO-MII (orange) împroves DPR's results (blue) by a margin. Note that for compressive Gaussian phase retrieval only unsupervised methods are tested, since cach mcasurement matrix requires retraining the model.
3.3 Ablation Studies
In our experiments, we observed that the performance can be improved by re-running the optimization procedure with a different initialization and selecting the result with the lowest magnitude error which is a common practice in image reconstruction. However, this approach quite costly. Our initialization schemes described Scction 2.2 can be used achieve a similar effect without the need of ro-rumning the whole optimization procedure multiple times. In Figure 4 we compare the single randomly initialized latent variable, 5000 initialization using the MII approach and slightly perturbed learned initialization (LI). Again, we consider a test set of 1024 samples. Although, are still using restarts in combination with our initialization, we observe that already for single run our initialization outperforms the results of runs.
In order to assess the impact of each step on the reconstruction performance, we perform an ablation study. We compare our complete PRILO-MIT and PRIIO-LII models with different modified versions of the model: for one, we consider omitting every optimization, i.c., we only compare with the initialization. Next, we omit the back-projection step (B) and the refmeeent step (C). We also compare with the variant of our model

---

Published in Transactions on Machine Learning Research (11/2022)
14
Table 3: Ablation experiments using StyleGAN on CelebA for Fourier measurements.
PSWA
13-
11
Method
PSNR (t)
SSTM (t)
PRILO-LI
23.3378
0.7247
1000
3000 Number intialiiaatinss
2000
4000
5000
only initialization
16.9070
0.4777
only step A only step A and B no ball constraint
23.0097
0.7108
22.9109
0.7063
22.8192
0.6974
PSNR
20
PRILO-MII
21.4223
0.6358
PNILO
only initialization only step A only step A and B no ball constraint
15.0566
0.3565
16
PRILO
20.1736
0.5861
14
19.9409
0.5719
2
Number
19.8461
0.5540
Figure Effect of the number of initializations and number of restarts onto the reconstruction performance. Shaded areas indicate the 95%- confidence interval.
which we only omit step C. Finally, we analyze the impact of the ball-constraints. Table shows that each of the components important to reach the results. Notably, only using step A and (which corresponds naïvely applying the approach by Daras al. to phase retrieval) gives worse results than only step A. However, in combination with our proposed step C the approach gives belter results. Figure in Appendix shows how omitting different steps affects the optimization.
4 Conclusion
Generative models play an important role in solving inverse problem. In the context of phase retrieval, we observe that optimizing intermediate representations is essential to obtain excellent reconstructions. Our method PRILO, used in combination with our new initialization schemes, produces better reconstruction results for Gaussian and Fourier phase retrieval than existing (supervised and unsupervised) methods. Notably, in some cases our unsupervised variant PRILO-MII even outperforms existing supervised methods. We also show that the initialization schemes introduced can easily be adapted to different methods for inverse problems that are based on generative models, e.g., DPR. Our ablation study justifies that each of our used components essential to achieve the reported results.
Limitations
During our experiments, we observed that some hyperparameter tuning necessary achieve good results: the selected intermediate layer highly influences the results and also the radií for the constraints play important role. Furthermore,
approach is in some cases not able to reconstruct the finer details and is still (to some exlend) limited by the generative model (FMNIST results reported in Table D. Furthermore, generator-based methods for non-oversampled Fourier phase retrieval still struggle to reconstruct high-resolution images demonstrated in Appendix
Broader Impact Statement
This work discusses a method for reconstructing images from their magnitude measurements. Whether this method have negative societal impacts is difficult to predict and depends the specific application, egg.,
10

---

Published in Transactions on Machine Learning Research (11/2022)
in X-ray crystallography or microscopy. For example, in some applications the reconstructed images could contain sensitive information, which could lead to data security issues if the method is not applied properly.
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 213-223. PMLR, 2017.
[2] Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference on Machine Learning, pp. 399-409. PMLR, 2020.
[3] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
[4] Emmanuel J Candes, Xiaodong Li, and Mahdi Soltanolkotabi. Phase retrieval via wirtinger flow: Theory and algorithms. IEEE Transactions on Information Theory, 61 (44: 2015.
[5] Mingqin Chen, Peikang Lin, Yuhui Quan, Tongyao Pang, and Hui Ji. Unsupervised phase retrieval using deep approximate mmse estimation. IEEE Transactions on Signal Processing, 00:2239-2252, 2022.
[6] Zhuojie Chen, Yan Huang, Yu Hu, and Zhifeng Chen. Phase recovery with deep complex- IEEE Signal Processing Letters, 29:887-891, 2022.
complex-domain domain priors.
[7] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pp. 2921-2926. IEEE, 2017.
[8] Giannis Daras, Josoph Dean, Ajil Jalal, and Alex Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In International Conference on Machine Learning, pp. PMLR, 2021.
[9] John Duchi, Shai Shalev- Shaler-Shwartz, Yoram Singer, and Tushar Chandra. Efficient projections onto the 11-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pp. 272 279, 2008.
10) C Fienup and Dainty. Phase retrieval and image reconstruction for astronomy. Image recovery: theory and application, 231:275, 1987.
[11] James Fienup. Phase retrieval algorithms: a comparison. Applied optics, 21(15):2758 2769, 1982.
[12] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. Advances in neural information processing systems, 30, 2017.
[13] Paul Hand, Oscar Leong, and Vlad Voroninski. Phase retrieval under generative prior. In Advances in Neural Information Processing Systems, pp. 9136 9146, 2018.
[14] Rakib Hyder, Viraj Shah, Chinmay Hegde, and M Salman Asif. Alternating phase projected gradient descent with generative priors for solving compressive phase retrieval. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7705-7709. IEEE, 2019.
[15] Rakib Hyder, Zikui Cai, and M Salman Asif. Solving phase retricval with a learned reference. In European Conference on Computer Vision, pp. 425 441. Springer, 2020.
[66] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improvod quality, stability, and variation. arXiv preprint arXim:1710.11 10196, 2017.
[17] Tero Karras, Samuli Laine, and Timo Aila. style-basod generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
11

---

Published in Transactions on Machine Learning Research (11/2022)
[18] Vinayak Killedar and Chandra Sekhar Seclamantula. Compressive phase retrieval based on sparse latent generative priors. In ICASSP 2022-2022 TEFF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1596 1600. LEEE, 2022.
[99 Hannah Lawrence, David A Barmherzig, Henry Li, Michael Eickenberg, and Marylou Gabrié. Phase retrieval with holography and untrained priors: Tackling the challenges of low-photon nanoscale imaging. Xiv preprint or urXiu:2012.07366 2020.
[20] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEFE, 86(11))2278-23)) 1998.
[21] Zhaoqiang Liu, Subhroshekhar Ghosh, and Jonathan Scarlett. Towards sample-optimal compressive phase retrieval with sparse and generative priors. Advonces in Neurol Information Processing Systems, 34, 2021.
[22] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[23] Raunak Manekar, Zhong Zhuang, Kshitij Tayal, Vipin Kumar, and Ju Sun. Deep learning initialized phase retricval. In NewrIPS 2020 Workshop on Deep Learning and Inverse Problems, 2020.
[24] Christopher Metzler, Phillip Schniter, Ashok Vecraraghavan, et al. prdeep: Robust phase retrieval with PMLR, 2018.
a flexible deep network. In International Conference on Muchine Learning, pp.
[25] Rick P Millane. Phase retrieval in crystallography and optics. JOSA A, 7(3):344-41, 1990.
[26] Arvind Neelakantan, Luke Vilmis, Quoe V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXin preprint ar arxi::511100660 2015.
[27] Yohei Nishizaki, Ryoichi Horisaki, Katsuhisa Kitaguchi, Mamoru Saito, and Jun Tanida. Analysis of non-iterative phase retrieval based on machine learning. Optical Review, 27(1):136-111, 2020.
[28] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
[29] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (rod). STAM Journal on Imaging Sciences, 1044):1804-1844, 2017.
300 Yoav Shechtman, Yonina C Eldar, Oren Cohen, Henry Nicholas Chapman, Jianwei Miao, and Mordechai Segev. Phase retrieval with application to optical imaging: a contemporary overview. IEEE signal processing magazine, 32(3))87-109, 2015.
[31] Niklas Smedemark- Margullie,, Jung Yeon Park, Max Daniels, Rose Yu, Jan-Willem van de Meent, and Paul Hand. Generator surgery for compressed sensing. arXiv preprint drx:::1002.11163 2021.
[32] Tobias Uelwer, Tobias Hoffmann, and Stefan Harmeling, Non-iterative phase retrieval with cascaded neural networks. In Artificial Neural Networks and Machine Learning ICANN 2021, pp. 295-306. Springer International Publishing, 2021.
[33] Tobias Uelwer, Alexander Oberstraß, and Stefan Harmeling. Phase retrioval using conditional generative adversarial networks. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 731-738 IEEE, 2021.
[34] Tobias Uelwer, Nick Rucks, and Stefan Harmeling, A closer look at reference learning for fourier phase retrieval. In NewrIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
[35] Adriaan Walther. The question of phase retrieval in optics. Optica Acta: International Journal of Optics, 10(1): (((1):41-48) 1963.
12

---

Published in Transactions on Machine Learning Research (11/2022)
[36] Gang Wang, Georgios B Giannakis, and Yonina C Eldar. Solving systems of random quadratic equations via truncated amplitude flow. TEEE Transactions on Information. Theory, 64(2):773-77) -94, 2017.
[37] Yaotian Wang, Xiaohang Sun, and Jason Fleischer. When deep denoising meets iterative phase retrieval. In International Conference Machine Learning, pp. 10007-10017. PMIR, 2020.
(38] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600 612, 2004.
[39] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681 688. Citeseer, 2011.
[4]] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1 708.0777, 2017.
[4]] Raymond Yeh, Chen Chen, Teck Yian Lim, Alexander Schwing, Mark Hasegawa-JJhnson, and Minh Do. Semantic image inpainting with deep generative models. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5485-5455 2017.
[42] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. Proceedings of the 1EEE conference on computer vision and poltern recognition, pp. 586 595, 2018.
[33 Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on compuler vision, pp. 592 608. Springer, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2022)
A VAE Architecture
For the MNIST, FMNIST and EMNIST dataset we use variational autoencoder (VAE) as generator. The encoder and the decoder cach consist of three layers with ReLU activation function, where cach layer had 500 hidden units. The latent space was chosen 100-dimensional. was trained using Bernoulli-ikelhhood and Adam with learning 10-3
3 for 100 epochs.
B CelebA Reconstructions using the DCGAN
PRILO
PRILO-MIT
PRILO-LI
Figure 5: Reconstructions from Fourier measurements on ColebA by PRILO and PRILO-MIT based on the DCGAN 28
C
CelebA Reconstructions using the Progressive GAN
PRILO
PRILO-MII
Figure 6: Reconstructions from Fourier measurements on CelebA by PRILO and PRILO-MII based on the
Progressive GAN
D Ablation Experiments: PSNR Throughout the Optimization
BC
BC
BC
PRILO-LI
PRILO-LI
PRILO-LI W/o
PSNR
2000
4000
6000
8000
Number steps
Figure 7: Ablation experiments for PRILO-LI based on StyleGAN on CelebA data: We omit different steps of our method and report the PSNR validation set consisting of 64 images. PSNR corresponding to the lowest magnitude error highlighted.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Table 4: Fourier phase retrieval: Our proposed method (being insupervised performs best among classical, unsupervised and supervisee reeeeiieed approaches for MNIST and EMNIST, and second best for FMNIST (in terms of PSNR). Best values are printed bold and second- second-bost b-bbsst are underlined. Numbers printed after are 95%%-confidence intervals calculated a batch of 1024 images.
0.5056 4 0.0113
0.4942 1: 0.0113
0.8568 + 0. 0.0097
0.9367 = 0.0091
0.0060
0.5616 # 0.0100
0.7598 + 0.0084
0.8537 1 0.0057
0.9416 1: 0. 0.0067
SSIM
0.9719
PSNR EMNIST
13.1157 1 0.2767
14.1934 + 0.4872
20. 10.9148 + 0.3219
33 8263 + 0.6440
37.1807 0.3257
14.4627 + 0.1614
17 40.409 + 0.1756
19.3163 1 0.1772
27. 1948 1 0,3873
0.3971 0.0122
0-4019 1 0.0L17
0.6846 + 0.0L13
0.7133 = 0.0120
0.0111
0.0109
0.0095
0.0090
1. 0 0.0098
SSIM (1)
0.7560
0.6070
0.7400
0.7768
0.8376
FMNIST
0. 1960
0.2002
= 0.2434
+ 0.3800
0.3864
17. 17.4787 0.1730
0.2165
0,2236
+: 0.4457
PSNR (1)
12. 22.7002
12. 72.9530
18. 8667
20 2082087
21 1834
20 2827
20 9069
25.5513
0.5504 + 0.0122
0.5274 4 0.0130
0.9070 + 0.0071
0.9843 + 0.0042
0.9949 + 0.0019
0.7292 # 0.0082
0.8191 + 0.0059
0.8529 1 0.0047
+ 0.0022
SSIM (1)
E Quantitative Results with Confidence Intervals
0.9800
MNIST
15.1 15.1820 1826 : 0.3782
20. 20.5696 # 1.9740
24.2118 + 0.3407
42 7584 + 0.5617
43 03.95411 + 0. 0.4554
17.0886 # 0.1838
18,5041 + 0. 1509
19.6216 1 0.1528
41 41.3767 1: 0.5584
PSNR (1)
E.1 MNIST, FMNIST and EMNIST
Learning
unsup.
unsup.
unsup.
sup.
sup.
sup.
sup.
MII (ours)
PRILO (ours)
PRILO MIILO-MIII
PRCCAN-L
Method
ResNet
IIIO
DPR
F2E
CPR
FR
15

---

Published in Transactions on Machine Learning Research (11/2022)
E.2 CelebA
Table 5: Fourier phase retrieval on CelebA: Our proposed PRILO combined with LI and StyleGAN achieves the best results in terms of PSNR and SSIM, also against enhanced version of DPR, which uses LI and StyleGAN. Best values are printed bold and second-best. are underlined. Numbers printed after + are 95%-confieence intervals calculated on batch of 1024 images.
CelebA
Method
Base model
Learning
PSNR (t)
SSTM (1)
ER
10.4036 + 0.1282
0.0637 = 0.0028
HIO
10.4443 + 0.1271
0.0510 + 0.0023
DPR
DCGAN
unsup.
16.9384 + 0.2534
0.4457 + 0.0124
DPR-MII
DOGAN
unsup.
17.8651 + 0.2485
0.4898 = 0.0123
PRILO
DCGAN
unsup.
18.3597 + 0.2781
0.5159 = 0.0136
PRILO-MIT
DCGAN
unsup.
18.5656 + 0.2431
0.5264 + 0.0119
DPR
Progressive GAN
unsup.
18.4384 + 0.3091
0.5276 + 0.0147
DPR-MII
Progressive GAN
unsup.
19.5213 + 0.2942
0.5738 + 0.0141
PRILO
Progressive GAN
unsup.
19.2779 + 0.3040
0.5665 + 0.0141
PRILO-MII
Progressive GAN
unsup.
19.9057 + 0.2894
0.5910 + 0.0137
DPR
StyleGAN
unsup.
18.2606 + 0.3745
0.4859 = 0.0181
DII-MII
StyleGAN
unsup.
20.3889 + 0.3279
0.5972 = 0.0154
PRILO
StyleGAN
wisup.
18.5542 1 0.3435
0.5143 1 0.0165
PRILO-MII
StyleGAN
wisup.
21.4223 1 0.2973
0.6358 1 0.0135
E2E
sup.
20.1266 + 0.1731
0.6367 + 0.0082
PROGAN-L
sup.
22.1951 + 0.2248
0.6846 + 0.0097
DPR-LI
StyleGAN StyleGAN
sup.
22.6223 + 0.2455
0.7021 + 0.0112
PRILO-LI
23.3378 sup.
+ 0.2691
0.7247 + 0.0120
F High-resolution Images
Table 6: Fourier phase retrieval high-rosolution images from FFHQ using a StyleGAN. Best values printed bold and second-best are underlined. Numbers printed after + are 95%-confidence intervals caleulated a batch of 128 images.
CelebA
Resolution
Method
Learning
PSNR (t)
SSIM (1)
DPR
unsup.
12.8724 + 0.7167
0.2942 + 0.0257
DPR-MII
unsup.
14.4224 + 0.6878
0.3468 #0.0270
256 256
PRILO
unsup-
12.9225 + 0.6873
0.3066 + 0.0252
PRILO-MII
unsup. 14.3080 + 0.7276
0.3722 #0.0282
DPR
unsup. 12.9071 + 0.6832
0.3673 + 0.0232
DPR-MII unsup. 14.4423 + 0.6957
0.4129 + 0.0223
512 512
PRILO
unsup. 12.9824 + 0.6939
0.3812 + 0.0237
PRILO-MII
unsup. 14.5264 + 0.6907
0.4562 + 0.0236
DPR unsup.
12.3979 + 0.6383
0.3891 + 0.0193
DPR-MII
unsup. 13.7563 + 0.6514
0.4888 + 0.0201
1024 1024
PRILO
unsup. 12.3938 + 0.6429
0.3953 + 0.0188
PRILO-MII
unsup.
13.8730 + 0.6577
0.4993 + 0.0197
16

---

Published in Transactions on Machine Learning Research (11/2022)
DPR
DPR-MIT
PRILO
PRILO-MII
Target
Figure 8: 512 512 reconstructions from Fourier measurements on FFHQ by DPR, DPR-MII, PRILO and PRILO-MII based on the StyleGAN.
17

---

Published in Transactions on Machine Learning Research (11/2022)
G
Hyperparameter Selection
Table 7: Hyperparameters for PRILO and PRILO-MII on the MNIST and EMNIST datasets.
Repetitions
Variable
Steps
f fi-radius
1
Initial optimization
20
150
100
Forward optimization
21
150
50
1
Back-projection
20
Refinement
20
Forward optimization
22
300
100
9
Back-projection
20
Refinement
50
Table 8: Hyperparameters for PRILO and PRILOO-MIT the Fashion-MNTST dataset.
Repetitions
Variable
Steps
fraadiss
Initial optimization
20
100
100
Forward optimization
21
100
10
1
Back-projection
20
Refinement
50
Forward optimization
22
200
40
Back-projection
20
Refinement
20
Table 9: Hyperparameters for PRILO (based on StyleGAN) on the CelebA dataset. Note that zo passed into each layer. That is why we have to apply erreguularization also for when optimizing the latter layers.
Repetitions
Variable
Steps
f-radius (2:)
l-aadius (zo)
Initial optimization
20
100
1000
1000
Forward optimization
100
200
100
Back-projection
50
100
1000
1000
Refinement
100
1000
1000
Forward optimization
22
100
200
100
Back-prooection
100
1000
1000
Refinement
100
1000
1000
18

---

Published in Transactions on Machine Learning Research (11/2022)
Table 10: Hyperparameters for PRILO-MIT and PRILO-LT (based on StyleGAN) on the CelebA dataset. Note that 2g passed into each layer. That is why we have apply 4-regularization also for 2g when optimizing the latter layers.
Repetitions
Variable
Steps
f1-radius (2))
L 1-radius (zo)
Initial optimization
20
2000
1000
1000
Forward optimization
400
100
100
Back- poock-rooection
50
100
1000
1000
Refinement
20
2000
1000
1000
19


Reviews:
[{"review_id": "hdk5QSUbwi", "reviewer": "Reviewer_J5zR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- S1: Meaningful gain in quality of generated images without additional supervision\n- S2: Proposed method is simple yet quite effective\n\nWeaknesses\n- W1: The proposed method is for phase retrieval but there is no analysis of the retrieved phases, instead the paper provides indirect analysis of the results -- evaluating the quality of generated images (probably via retrieved phases)\n- W2: While the method is proposed for both Fourier and Gaussian phase retrieval, most of the analyses are done in Fourier phase retrieval (except the Fig 3). Is there anything special in this method working better in Fourier phase retrieval?\n- W3: Some presentations are not clear (see requested changes)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for phase retrieval by (1) optimizing intermediate representations (modifying intermediate layer optimization, ILO, method), named PRILO, and (2) two initialization schemes. To validate the method, the paper uses two examples of Fourier phase retrieval and compressive Gaussian phase retrieval as an implicit application, and shows improves generated image quality as the performance measure. The proposed method does not require additional human supervision, thus unsupervised method, and improves the quality of generated images in noticeable gains (large gain in MNIST variant datasets, meaningful gain in CelebA dataset).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- In abstract, \"generative models are a new idea to constrain the solution set\" is not clear. Specifically, how does the generative model constrain the solution set?\n- In 2.1. B. Back-projection section, why this serves as a regularizer for the solution by step A?\n\t- Isn't z_0 in this section \\bar{z_0}?\n- The reviewer encourages to empirically compare the \"random restarts and to select the solution with the lowest magnitude error after the optimization\" (in Sec. 2.2) to the proposed method.\n\t- Likely in Figure 4\n\t- Also citation is needed for this sentence\n- Please explain why you only evaluate LI initialization on Fourier phase retrieval.\n- Please explain why why step C (the proposed method) is not very effective in StyleGAN on CelebA for Fourier measurements (Table 3 - PRILO-LI section)\n- In Table 1, the latest method compared is DPR [12], which is published in 2018. We have list of relevant work as follows to name a few:\n\t- Kiledar et al., Compressive Phase Retrieval Based On Sparse Latent Generative Priors, ICASSP 2022\n\t- Chen et al., Unsupervised Phase Retrieval Using Deep Approximate MMSE Estimation, IEEE Transactions on Signal Processing, April 2022\n\t- Chen et al., Phase Recovery With Deep Complex-Domain Priors, IEEE Signal Processing Letters, March 2022\n- In all equation reference, \"Equation equation ??\" should be \"Equation ??\"\n- Figure 1 in text (C. Refinement in Sec. 2.1) has no link.", "broader_impact_concerns": "No much concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pNA3ZvCfuH", "reviewer": "Reviewer_iiKt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\n1. The paper is relatively well-written.\n2. This paper considers an important and interesting application of generative models.\n3. The ablation studies are detailed.\n\n**Weakness:**\n1. Limited theoretical insights. The proposed approach focuses more on the engineering side (e.g., feature engineering, combining different training losses). The approach is not theoretically justified and appears to rely heavily on parameter tuning.\n2. In section 1.4, there are only claims but without analysis, insights, or evidence. The arguments can be tenuous without proper justification. \n3. The training objective in equation (9) is not properly justified in the phrase retrieval context. It also appears similar to the GAN inversion objective. It would be good to explicitly explain why each term is necessary, why it is a valid objective function and how you select the weight $\\lambda$. At the same time, the learning objective seems to be hand-crafted without too many theoretical insights. \n4. Need to compare with stronger baselines. It would be good to compare with more recent (learning-based) unsupervised baselines in Table 1. As pointed out in 3.1, baseline [9] and [10] do not have learning components. So it is not surprising that the proposed learning-based method can easily outperform the two baselines.\n5. For the empirical results, it would be good to report the error rates (e.g., averaged performance over multiple runs/random seeds).\n6. The image datasets considered here are relatively low-resolution. It would be good to consider higher-resolution datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a generative model based approach for phase retrieval. Specifically, the authors design an intermediate layer optimization approach and introduce an initialization scheme that further improves the data reconstruction quality. Extensive empirical results show the effectiveness of the approach on certain datasets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. One major concern is the technical novelty of the work. The proposed approach relies heavily on feature engineering and parameter tuning. There is very limited novelty on the technical side and it also remains unclear where the performance boost comes from. It would be good to provide some extra discussion and theoretical insights. In terms of the experiment, it would be good to report averaged results using different random seeds. Since the contribution is mainly on the empirical side, it would be good to also consider higher-resolution image datasets.\n\n2. The proposed model framework is similar to a hierarchical generative model. It would be good to add some discussion with hierarchical generative models (e.g., hierarchical VAE).\n\n3. The parameters should be formally defined. For instance, in eq (4), it would be good to define $r$ (e.g., a positive scalar) first. In section 2, should it be $G_1^{k}=G_{i+1}^{k}\\circ G_{1}^{i}$?\n\n4. At the same time, section 1.4 seems to be tenuous. Instead of having two claims, it would be better to provide more intuition or analysis to convince the readers why that is the case.\n\n5. It would be important to provide more analysis and elaborate on why eq (9) is a good objective function. It also seems that the performance will depend heavily on the selection of the weighting constant $\\lambda$. It would be good to elaborate on how you select the weights and provide some theoretical analysis (if possible). How robust the weights are for different datasets?\n\n6. It would be better to also include stronger and more recent unsupervised baselines in Table 1.\n\n", "broader_impact_concerns": "This work proposes an approach to reconstructing data based on their magnitudes. In many applications, the reconstructed data could contain sensitive information. There could be a negative societal impact (e.g., data security issue) if the approach is not used properly. It would be good to expand the discussion in the \"Broader Impact Statement\" section more.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C2Ql4EL1nG", "reviewer": "Reviewer_8SrU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n## Strengths \n\n* The paper is very well written, and the ideas are presented in a clear manner \n* The paper addresses a classic problem that is of interest to various imaging applications with modern generative models\n* The experiments clearly delineate the contributions of the proposed steps.  Also, the effect of the proposed initializations is tested not only on the proposed method, PRILO, but also on the DPR method [Hands et al. 2018]  showing a consistent improvement in performance.\n* The experiments were conducted on various neural architectures showing consistent gains independent of the architecture choice\n* The experiments were conducted with both supervised and unsupervised learned priors\n\n## Weaknesses \n\nWhile I do not see major weaknesses in this paper, below are a few points that stood out to me. \n\n* While the experiments in the paper are great, they are on datasets that do not reflect the actual applications of phase retrieval. Importantly, the proposed approach relies on the assumption of having a good generative model of the images of interest. However, constructing such generative models in applications relevant to phase retrieval (e.g., crystallography or CDI) is not trivial.   \n\n* Minor Typo(s):\n  * Page 3 - \"Equation equation 3\"\n\n## Overall Assessment\n\nTo summarize my review, I think the paper is well written, and it discusses an important problem that is of interest to many imaging applications. I did not find any technical flaws in the paper, and the paper's experimental section supports the authors' claims. Overall, this is a good paper with sufficient technical contributions, nice experimental section, and a simple and direct narrative. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary \n## Background\nThe paper focuses on the phase retrieval problem, a specific nonlinear inverse problem with various applications from crystallography to coherent diffraction imaging (CDI), and leverages deep generative models with intermediate layer optimization (ILO) to solve this inverse problem. Briefly, given only the Fourier magnitudes, the goal is to reconstruct an image (i.e., retrieve the phase); however, this problem is ill-posed, as the Fourier magnitudes do not uniquely identify the phase. A possible solution is to learn image priors and constrain the solution space to the learned priors, which one can do with generative models. Therefore, to solve the inverse problem, one can optimize over the domain (i.e., the latent space) of a generative model to generate an image in the range of the model with Fourier magnitudes that match the observation. \n## Contributions\nIn this work, the authors show that by leveraging intermediate layer optimization (ILO), one can significantly improve this process and the quality of reconstructed images. In particular, the authors propose an iterative process involving three steps. In the first step, the authors perform a constrained optimization on the intermediate layers (constrained to a $\\ell_1$-ball), which they call forward optimization. In the second step, they back-project the optimized intermediate representation to update the latent code in the input domain of the generator. These two steps have also been used in prior work. However, the authors propose a third step, in which they refine the latent code through another constrained optimization. Importantly, inverse problem optimization is highly non-convex, and a good initialization plays a vital role in obtaining a satisfactory performance. To that end, the authors propose two alternative initialization schemes, namely Magnitude Informed Initialization (MII) and Learned Initialization (LI), through a learned encoder (similar to GAN inversion models). Lastly, the proposed approach is tested on Fourier and Gaussian phase retrieval problems on MNIST, FashionMNIST, EMNIST, and CelebA datasets showing improved performance over existing methods. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* There are several implementation details that I missed in the paper. \n  * How is the intermediate layer, i.e., layer $i$, selected to perform the optimization? Is this done sequentially? \n  * The three steps proposed in the paper could be applied iteratively and on different layers. What is the effect of the choice of the sequence of layers to optimize? \n\nIncluding these details in the paper could further increase the paper's impact.", "broader_impact_concerns": "The broader impact is sufficiently discussed by the authors. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I judge that the paper satisfies the main two criteria of (i) supporting the claims made, and (ii) being of interest to some TMLR readers.  The reviewers all opted for 'leaning accept', so none were strongly in favor, but at least all were in favor.  Some concerns remained about reproducibility and lack of realistic phase retrieval applications, but I don't view these as deal-breakers.
Decision: Accept (poster)