Title: PLUGIn: A simple algorithm for inverting generative models with recovery guarantees
Authors: Babhru Joshi
Xiaowei Li
Yaniv Plan
Ozgur Yilmaz
Venue: NeurIPS 2021
Keywords: generative neural network
inverse problem
denoise
gradient type method
non-convex optimization

Abstract: We consider the problem of recovering an unknown latent code vector under a known generative model. For a $d$-layer deep generative network $\mathcal{G}:\mathbb{R}^{n_0}\rightarrow \mathbb{R}^{n_d}$ with ReLU activation functions, let the observation be $\mathcal{G}(x)+\epsilon$ where $\epsilon$ is noise. We introduce a simple novel algorithm, Partially Linearized Update for Generative Inversion (PLUGIn), to estimate $x$ (and thus $\mathcal{G}(x)$). We prove that, when weights are Gaussian and layer widths $n_i \gtrsim 5^i n_0$ (up to log factors), the algorithm converges geometrically to a neighbourhood of $x$ with high probability. Note the inequality on layer widths allows $n_i>n_{i+1}$ when $i\geq 1$. To our knowledge, this is the first such result for networks with some contractive layers. After a sufficient number of iterations, the estimation errors for both $x$ and $\mathcal{G}(x)$ are at most in the order of $\sqrt{4^dn_0/n_d} \|\epsilon\|$. Thus, the algorithm can denoise when the expansion ratio $n_d/n_0$ is large. Numerical experiments on synthetic data and real data are provided to validate our theoretical results and to illustrate that the algorithm can effectively remove artifacts in an image.


Review discussion:
Inverting generative models is a very clean and well-motivated algorithmic challenge, and this paper contributes many interesting new ideas for this problem. From the point of view of the algorithm, existing approaches for this problem rely on showing that that the landscape for the square loss L(x) = ||G(x) - y||^2 is benign with high probability for expansive networks, so that gradient descent with respect to L converges to an approximate inverse for y. In this context, it is quite innovative to study the alternative update rule in this paper, and the fact that it works just as well for this problem and can even handle non-expansive networks is very interesting.
From the point of view of the analysis, existing works are based on verifying that a tall and skinny Gaussian weight matrix whp satisfies a certain "weight distribution condition" which one could also interpret as a bound on the supremum of some random process, this is by definition a condition that must hold *layer-by-layer* for the Hand-Voroninski-style analysis to apply.
In contrast, the random processes that this paper considers are defined in terms of *sub-networks* of G rather than individual layers, which is why one can hope to get away with "expansivity on average" rather than strict expansivity.
Additionally, the technical details are well-written, and the example calculations in the first nine pages give the reader a good sense for how their proof works at a high level. I would enthusiastically recommend acceptance.
Minor comments:
- What are the obstacles for handling the setting where one only has compressive measurements of y?
- Caption for Figure 2 should be self-contained, i.e. should write that solid line is PLUGin and dotted line is gradient descent.
- In the display below line 462 of supplement, fix formatting for absolute values in |\sum^m_{i=1} X_i|
- Line 517 of supplement: capitalize "proof"
- Line 559 of supplement: "result" -> "results"
No
We thank the reviewer for their review and comments. Please find our responses below:
**Compressive measurements:** A result for the compressive measurement case can be readily obtained using similar techniques as the ones provided in the paper. In this case, we would consider the activation function of the last layer to be the identity function and the weight matrix of the last layer to be the compressive measurement matrix. In the random case, the recovery result would require a sample complexity on the order of $O(5^dn_0)$, up to log factors. Here, $n_0$ is the dimension of the latent code vector and $d$ is depth of the network. We believe, as an artifact of the proof technique, this sample complexity is optimal in $n_0$ but sub-optimal in $d$. We leave improving this dependence on $d$ to future work.
**Typos and Caption:** We will fix them in the revised paper. Thank you again for pointing them out.
Thanks for clarifying regarding compressive measurements! I continue to believe that this is a very strong paper and enthusiastically recommend acceptance.
I think this is a good paper and makes progress in understanding when and how generative models are invertible. Prior work had intermediate layers with diagonal matrices, and could not tolerate noise that was only independent of the weight matrices.
I have read the author response.
No
We thank the reviewer for their review.
Unfortunately, I don't familiar quite well with generalized Dudley’s integral theory, so the questions are about numerical experiments:
`1. Did you try to check your method for other image datasets (for example CIFAR10)?
2. Figures 2c and 2d show that gradient descent numerically faster converges compared with PLUGIn. Does your method work much faster than SGD? Evaluations on time comparison for image/synthetic datasets and additional comments on the performance of PLUGIn compared with SGD are recommended.
No
Thank you for your review and suggestions. Please find our responses below:
**Experiments on image dataset besides MNIST:** The paper is primarily focused on theoretical exposition of the PLUGIn algorithm. For numerical experiments, we provide preliminary positive results on its performance on MNIST dataset. We agree that further experiments on other datasets would be beneficial in demonstrating the applicability of the PLUGIN algorithm. We leave this to future work.
**Comparison to SGD:** A natural choice of program to apply the stochastic gradient descent algorithm, in the context of the inverse problem considered in the paper, is
$$ \\min_{x\\in\\mathbb{R}^{n_0}} \\|y - \\mathcal{G}(x)\\|_2^2,$$
which is equation (3) in the paper.
However, the objective in the above problem is not separable in entries of $x$ because the generator $\\mathcal{G}$ is a fully-connected neural network.
As a result, we don't believe SGD can be efficiently implemented in this context and so a comparison to SGD is not provided.

Meta-review:
role: PC
data: comment: This paper studies the problem of inverting a generative model. More precisely, we are given a ReLU network $G$ and we observe $y = G(x^*) + \epsilon$ and the goal is to recovery $x^*$. The paper gives strong provable guarantees under much milder assumptions than in earlier work. Earlier work of Hand and Voroninski gives recovery guarantees when the weights matrices in the network are Gaussian and each layer is expansive. This paper is able to weaken the former condition to an expansion on average. In particular some layers can be contractive, as long as the ratio of their output to the size of the input to the network is large. Further they give a simple iterative algorithm. The intuition behind it is clear, however the technical analysis is complicated by the fact that the iterates depend on the randomness. All the reviewers agreed that this paper takes a definitive step forward on an important class of problems. 

Decision: Accept (Spotlight)