Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work considers the problem of neural network inversion, which lies at the core of many downstream tasks incorporating neural networks, such as inverse problems. Naively inverting a network via gradient descent is challenging, due to the induced non-convex landscape with potentially many local minima. The authors present a method that takes a pretrained neural network $F$ and learns an encoder $\Theta$ such that gradient descent is easier to perform over the input space of $F \circ \Theta$, as opposed to directly over the input space of $F$. The model is trained by minimizing a loss over many trajectories of gradient descent for a fixed number of steps. The authors apply their approach to GAN inversion, 3D pose reconstruction, and protecting against adversarial attacks. The proposed approach shows that gradient descent empirically requires fewer steps to approach a solution as compared to direct inversion.
strength_and_weaknesses: **Strengths:**
- The proposed training strategy allows for gradient descent to be performed more quickly in the latent space in that reasonable solutions are found with fewer iterations.
- The new encoder performs better on out of distribution images for inversion than standard encoder-based inference approaches.

**Weaknesses:**
- The way the methodology is proposed, the encoder would be specific for the forward model considered, rather than simply to the GAN or generator being inverted. I wonder if the authors could comment on a scenario that could help broaden the methods scope (in the next section).

clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper was well-written and easy to follow.

**Novelty:** To the reviewer's knowledge, incorporating gradient descent trajectories in the training of an encoder for neural network inversion is novel.

**Reproducibility:** No code is provided, but training and implementation details are provided.

**General comments:**
- In regards to one of my earlier comments about the applicability of the method, I had a question about the following scenario. One useful thing about typical GAN priors are that they are agnostic to the forward model (given measurements $y = f(x) + \eta$, one could use a GAN $G$ to reconstruct $x$ by solving $\min_z ||f(G(z)) - f(x)||$ by gradient descent). Thus, in principle this GAN could be used for different forward models $f(\cdot)$ and solve different inverse problems (over the same image distribution). The proposed method, however, learns an encoder so that optimization over the input of $f\circ G$ is easier. This also requires examples of outputs $y$. Could one, instead, train the encoder to be GAN specific and then use it for multiple forward models down the line? For example, could one train the encoder to invert a StyleGAN with gradient descent and then be used at inference time to solve an inpainting or deblurring problem? 
- How does the size of the input space of the encoder play a role in the results? For example, does having a lower latent dimension in the encoder aid/hurt the results (potentially by having less representational capacity)?

summary_of_the_review: Overall, I found the ideas in the paper intuitive and the results look fairly compelling.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework to accelerate and stabilize the process of neural network inversion. Specifically, the proposed method uses a neural network to learn a mapping between the latent spaces that allows gradient descent over the input to converge in fewer steps. Experiments show that the proposed framework can successfully improve the convergence speed for optimization.
strength_and_weaknesses: ### Strength
* The paper is well-written and easy to follow. 
* The motivation is clear, and the intuition behind the proposed method is straightforward.
* The experiment results seem promising. Inverting a neural network has a lot of advantages and downstream applications. The proposed * method appears effective and can further accelerate and stabilize such a process.
### Weaknesses
My primary concern is regarding the sub-optimal solution when using the proposed method. Is there any circumstance of Z that may result in a suboptimally averaged solution since the loss landscape is a smoothed-out version of X? If so, it is reasonable, as it can be considered a tradeoff between speed and quality. However, showcasing such examples and quantitatively measuring the tradeoffs would be better.
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is clearly written and easy to follow in general.
### Novelty
The proposed method is marginally novel, using a neural network as a surrogate of latent space.
### Reproducibility
Codes are not provided, but sufficient implementation details are given.
summary_of_the_review: I think this is an interesting paper to accelerate neural network inversion. Although the proposed method is rather simple, it seems to be effective. I am willing to raise my recommendation if the question mentioned in the weakness section is addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a method that learns a loss landscape
where gradient descent is efficient, bringing massive improvement and acceleration
to the inversion process of learning a CNN. Advantages are demonstrated using a number of methods
for both generative and discriminative tasks.

The method helps to accelerate optimization-based inference to invert a forward model. The concept involves
learning a new space that is easier than the original input space to optimize with gradient descent at testing time.

strength_and_weaknesses: 
Paper is easy to follow and read. Enough of empirical results provided to substantiate the claim.


Algorithm 1 is not referred in text; although trivial matter - bit it should be done at least once.
"MAML" is not specified anywhere in document- pl, expand the term.

Is your method similar to applying ADMM over a manifold/topological space, to optimize ?

Would have preferred more analytics concerning the convergence of your algorithm  - does that have any minimal guarantee ?

You talk of patterns of trajectories being learned from latent space.
 How many such pattern clusters exist? DO they sufficiently represent categories of populations required
with no bias?


Few references not cited, some example:
Landscape and training regimes in deep learning; Mario Geiger, Leonardo Petrini, Matthieu Wyart; Physics Reports, Volume 924, 15 August 2021, Pages 1-18.

Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment; Ziqi Yang et al; Proceedings CCS '19; Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment; https://dl.acm.org/doi/abs/10.1145/3319535.3354261

T. Hospedales, A. Antoniou, P. Micaelli and A. Storkey, "Meta-Learning in Neural Networks: A Survey" in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 44, no. 09, pp. 5149-5169, 2022.
doi: 10.1109/TPAMI.2021.3079209

TENGRAD: TIME-EFFICIENT NATURAL GRADIENT DESCENT WITH EXACT FISHER-BLOCK INVERSION; Saeed Soori et. al; 
https://www.cs.toronto.edu/~mmehride/papers_private/TENGRAD.pdf





clarity,_quality,_novelty_and_reproducibility: 
Concerned about incremental novelty.

Most of the paper appears to be reproducible, although unless someone must experiment with the
code, else it may be difficult to completely judge.

summary_of_the_review: 

Inversion of FF models will help solving many inverse problems.
The proposed method may help researchers to explore more work in this direction.

The lack of sufficient analytics is a major cause of worry.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores optimization-based neural network inversion. Instead of optimizing in the original input space, they propose to learn a new latent space and run optimization in the learned space during test time. They show that their technique greatly accelerates the inversion optimization process for a variety of tasks including GAN inversion, adversarial defense, and human pose reconstruction.
strength_and_weaknesses: **Strength:**
1. The proposed technique is very simple yet it has been shown to be effective (greatly speeding up the inversion process) on several tasks involving neural network inversion.
2. The writing is easy to follow. The illustrations are helpful for understanding. The mathematical and experimental details are clear to me.

**Weaknesses:**
More empirical studies need to be done in order to support their claim and fully understand the proposed technique:
1. Firstly, does the learned latent space only depend on the model architecture, not the data? To what extent can we generalize out of domain? Can it be used to accelerate inference using the same model but with a *very* different dataset? I understand that the authors have already conducted experiments in the OOD scenario, but it would convince me even more if we can see the whole spectrum going from very similar datasets to completely different datasets (This analysis can be done using even synthetic data in a controlled setting). In addition, I would even suggest using random inputs to generate data and learn the latent space and run inferences using real data. It would be very interesting if we could still observe any improvement at all.
2. Secondly, it would be better if we could understand how much the learned latent space depends on the model architecture. For the purpose of analysis, what if we use the same dataset and a different model architecture? 

I am not fully convinced by their statements regarding:
1. They mentioned that OBI supports generating diverse hypotheses while EBI can only generate one outcome. I think this might not be true as you can always inject noise into the encoder to generate diverse samples. One can also parameterize their encoders to output distributions over inputs.
2. They stated in the second paragraph that OBI can adapt without additional training. Will that still be true with the proposed learned new latent space or will the performance drop (at least slightly) with partially missing observations or new data distributions?
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
The paper is relatively well-written and has enough technical details. The illustrations are helpful and the presentation of results is clear.

**Quality:**
The technical part is simple and clear enough. It would be a plus if they can provide some theoretical explanation about why the proposed technique can learn better loss landscape (but I don't think the authors need to work on that during the rebuttal). The experiments include a wide range of problems and the results are interesting. However, I am not fully convinced because of the reasons stated in the "weaknesses" part.

**Novelty:**
Although the high-level idea of learning better latent space for optimization has also appeared in other subareas (e.g. [Meta-Learning with Latent Embedding Optimization](https://arxiv.org/abs/1807.05960)), applying this idea to neural network inversion is new as far as I know. The problems they are trying to solve are important because they are ubiquitous in applications.

**Reproducibility:**
The authors promised in the main text that they will release their code and model. No code has been provided yet.
summary_of_the_review: I recommend weak acceptance of this paper. On the one hand, this paper proposes a very simple technique that solves an important problem and they have applied their approach to a sufficient number of problems. On the other hand, I don't think I fully understand how their approach works and I think there should be more study (either theoretical justification or empirical study as suggested in the "weakness" section).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work considers the problem of neural network inversion, which is an important problem in the field of machine learning (for instance, it is quite popular for inverting GANs). The authors claim that naive inversion with gradient descent might be prone to having difficulties due to highly non-convex landscapes. They present a method that takes a pretrained neural network and learns an encoder such that gradient descent is easier to perform over the transformed space defined by the encoder.

The reviewers were mildly positive about the paper (6,6,6,5) and did not change their opinion after the discussion period. This paper was therefore discussed with the meta-AC and we concluded this paper is not yet ready for acceptance.

Although the paper presents some encouraging empirical results, there are several important problems that remain:
- The paper does not provide any strong evidence explaining why the proposed mapping yields a more efficient method. It is for instance not clear whether Figure 2 is accurate in any way. The only justification is a set of PCA visualizations of the landscape in Appendix A, but this is very preliminary and incomplete evidence.

- The paper ignores a large body of literature that enforces smoothing constraints on the latent space of probabilistic models, see for instance the seminal paper by Tipping and Bishop:
Tipping, Michael E., and Christopher M. Bishop. "Probabilistic principal component analysis." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 61.3 (1999): 611-622.
http://www.cs.columbia.edu/~blei/seminar/2020-representation/readings/TippingBishop1999.pdf

- This is more minor but there is a lot of work in the field of optimization that shows that SGD can optimize complex non-convex landscapes, I would also expect a deeper discussion in the paper.

In summary, I'm afraid I'm not able to recommend acceptance at this stage. I strongly encourage the authors to address these two main points before resubmitting their paper to another venue.
justification_for_why_not_higher_score: See above comments. Reviewers mildly positive, this case was discussed with the meta AC and we decided to reject the paper for now.
Decision: Reject