Landscape Learning for Neural Network Inversion
Authors: Ruoshi Liu
Chengzhi Mao
Purva Tendulkar
Hao Wang
Carl Vondrick
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Network Inversion
Loss Landscape
Optimization

Abstract: Many machine learning methods operate by inverting a neural network at inference time, which has become a popular technique for solving inverse problems in computer vision, robotics, and graphics. However, these methods often involve gradient descent through a highly non-convex loss landscape, causing the optimization process to be unstable and slow. We introduce a method that learns a loss landscape where gradient descent is efficient, bringing massive improvement and acceleration to the inversion process. We demonstrate this advantage on a number of methods for both generative and discriminative tasks, including GAN inversion, adversarial defense, and 3D human pose reconstruction.

Under review as a conference paper at ICLR 2023
LANDSCAPE LEARNING FOR NEURAL NETWORK INVERSION
Anonymous authors Paper under double-blind review
ABSTRACT
Many machine learning methods operate by inverting a neural network at inference time, which has become popular technique for solving inverse problems in com- puter vision, robotics, and graphics. However, these methods often involve gradient descent through a highly non-convex loss landscape, causing the optimization process to be unstable and slow. We introduce method that learns loss landscape where gradient descent is efficient, bringing massive improvement and acceleration to the inversion process. We demonstrate this advantage number of methods for both generative and discriminative tasks, including GAN inversion, adversarial defense, and 3D human pose reconstruction.
INTRODUCTION
Many inference problems in machine learning are formulated as inverting forward model F(x) by optimizing an objective over the input space This approach, which term optimization-basee inference (OBI), has traditionally been used to solve a range of inverse problems in vision, graphics. robotics, recommendation systems, and security (Hernandez et al. 2008 Lee Kuo 1993; Domke 2012, Brakel et al. 2013 Stoyanov et al. 2011 Cremer et al. 2019). Recently, neural networks have emerged as the paramterization of choice for forward models (Loper al. 2015 Pavlakos
Abdal Abdal et al., 2019 Menon et al., [0200] Yu et al. [2021] Wang et al.] 2019 Chen et al. 2022
Zhang et al. 202 I, which can be pretrained on large collections of data, and inverted at testing time
in order to solve inference queries.
Optimization-basse inference (OBI) has a number of advantages over feed-forward or encoder-based inference (EBI). Since there is encoder, OBI provides flexibility adapt to tasks, allowing one to define new constraints into the objective during inference. When observations are partially missing, OBI can adapt without additional training. Moreover, OBI naturally supports generating multiple and diverse hypotheses when there uncertainty. Finally, OBI has intrinsic advantages for robustness, both adapting new data distributions well defending against adversarial examples.
However, the key bottleneck for OBI in practice is the computational efficiency and the speed of inference. Feed-forward models fast because they only require single forward pass of neural network, but OBI requires many (often hundreds) steps of optimization in order to obtain strong results for one example. Forward models in OBI are often trained with generative or discriminative tasks, but they are not trained for the purpose of performing gradient descent in the input space. Fig. g visualizes the loss landscape for uncurated examples. The loss landscape not guaranteed be an efficient path from the initialization to the solution, causing the instability and inefficiency.
In this paper, we propose framework accelerate and stabilize the inversion of forward neural networks. Instead of optimizing over the original input space, we will learn new input space such that gradient descent converges quickly. Our approach uses an alternating algorithm in order to learn the mapping between these spaces. The first step collects optimization trajectories in the new space until convergence. The second step updates the mapping parameters to reduce the distance between the convergence point and each point along the trajectory. By repeating these steps, approach will learn a mapping between the spaces that allows gradient descent over the input to converge in significantly fewer steps.
Empirical experiments and visualizations on both generative and discriminative models show that method can significantly improve the convergence speed for optimization. We validate our

---

Under review as a conference paper at ICLR 2023
approach on diverse set of computer vision tasks, including GAN inversion (Abdal al. 2019). adversarial defense (Mao et al. 2021), and 3D human pose reconstruction (Pavlakos et al. (019). Our experiments show that method converges order of magnitude faster without loss absolute performance after convergence. As our approach does not require retraining of the forward model, it can be compatible to all existing OBI methods with differentiable forward model and objective function.
The primary contribution of this paper is an efficient optimization-base inference framework. In Sec. 2 we survey the related literature to provide an overview of forward model inversion problem. In Sec. 3 we formally define OBI (3.1); our method to learn an efficient loss landscape for OBI (3.2); a training algorithm for better generalization and robustness 3.3 In Sec. 4 we experimentally study and analyze the effectiveness of the mapping network for OBI. We will release all code and models.
RELATED WORK
The different approaches for inference with a neural network can be partitioned into either encoder- based inference, which is feedforward, or optimization-based inference, which is iterative. We briefly review these two approaches in the context of our work. addition, we also review representative work in meta-learning and discuss the similarities and differences with our work.
2.1 RECODER-BASER INFERENCE
Encoder-based inference trains a neural network F' to directly map from the output space to the input space. Auto-encoder based approach (Pidhorskyi et al., 2020) learns encoder that map the input data to the latent space. Riihardsson et al. (2021); lov et al. (2021); Wei et al. 202T); Perarnau et al. (016)) learn an encoder from the image the latent space in GAN. Encoder based inference requires training the encoder on the anticipated distribution in advance, which is often less effective and can
fail on unexpected samples (Dinh al. 2021 Kang al. 2021)
2.2 OPTIMIZATION-BASEEI INFERENCE
OBI methods perform inference by solving an optimization problem with gradient-based methods such as Stochastic Gradient Descent (SGD) (Robbins & Monro
|onro|[95[] 195I) and Projected Gradient Descent (PGD) (Madry et al. 2017). In these cases, the objective function specifies the inference task. Besides these methods which use point estimate for the latent variable, one can estimate the posterior dis- tribution of the latent variable through Bayesian optimization, such as SGLD (Welling & Tch 201
Gradient based optimization methods have been used to infer the latent code of query samples in deep generative models like GANs (Goodfellow et al.) 2014) via GAN inversion (Karras al.
2020
Jahanian et al. 2019 Shen et al.; 2020 Zhu et al. 2016; Abbda et al,, 2019, 2020) Bau et al.
2019 Huh et al., 2020 Pan al. 2021: Yang al. 2019). Style transfer relies gradient based optimization to change the style of the input images Jing et al. (219)) It can also create adversarial attacks that fool the classifier (Croce & Hein 2020; Carlini & Wagner] 2017: Mao et al. 2019
Szegedy et al.) 2013). Recently, back bock-propagation based optimization has shown to be effective defending adversarial examples (Mao al. (021) and compressed sensing (Bora al. 2017).
Wu et al. (2019) uses Model-Agnosee Meta-Learning (MAML) (Finn et al. 2017) to accelerate the optimization process in compressed sensing.
Recently, constrained optimization was popularized for text-to-ing sot-to-image synthesis by Crowson et al. (2022) : Liu al./ (202T). They search in the latent space to produce an image that has the highest similarity with the given text as measured by multi-modal similarity model like CLIP (Radford) al. 202T). Test-time constrained optimization is also related to the idea of 'prompt-tuning for large language models. Lester et al. (2021) learns "soft prompts" to condition frozen language models to perform specific downstream tasks. Soft prompts are learned through backpropagation to incorporate signal from just few labeled examples (few-shot learning).
major challenge for optimization- based inference is how to perform efficient optimization in a highly non-convex space Geiger al. (2021). To address this, input convex model (Amos et al.) 2017) was proposed so that gradient descent can be performed in a convex space. Triip et al. (2020) introduced a method retrain the generative model such it learns latent manifold that is easy
2

---

Under review as a conference paper at ICLR 2023
Do
F
L
8
Z
X
Figure 1: Method. The left and middle figure show the loss landscape for latent space and the original latent space, respectively. While walking to the optimal solution few steps is hard X space, it can be done in our learned loss landscapes.
to optimize. When the model cannot be changed and retrained, bidirectional inference (Wang al. 2019) and hybrid inference (Zhu et al. 2016; 2020) uses encoder provide good initialization for the optimization-based inference in a non-convex space. (Sooi et al. (202T) proposes eine-ffficient Natural GRadient Descent method for solving inverse of neural network's Fisher information matrix. Our method does not retrain the generative model, but instead maps new latent space into the original input space, allowing more efficient optimization.
2.3
META-LEARNING
Given distribution of tasks, meta- meta-learning aims to adapt quickly when presented with previously unseen task Hospedales et al. (2021). MAMI (Finn et al. 2017) and related methods (Finn et al. 2018 2019) propose a method to learn parameter initialization of neural network by differentiaing through the fine-tuning process. To reduce the computational cost of MAML due to the 2nd degree gradient, Nichol et al. (2018) proposes first-order meta-learning algorithm. Unlike MAML which include modifying the forward model (e.g. in Wu et al. (2019), our approach is able to keep the forward model fixed in order to maintain its learned rich priors. Instead of meta-learning of model initialization, a line of work (Andrychowicz et al., 2016: Ravi & Larochelle 2016 Chen et al. 202Tb Wichrowska et al. 2017) proposed to learn an optimizer, often in form of an RNN, in order to accelerate gradient descent. Unlike learned optimizers that try to create better optimization algorithms, our approach instead learns to remap the loss landscape which is compatible with any choice of optimizer, including standard SGD or learned ones.
3 LEARNING LANDSCAPES FOR EFFICIENT INFERENCE
We present our framework to learn efficient loss landscape for optimization-basel inference (OBI) methods. Sec. 3.1 we will define OBI. In Sec. 3.2, we will introduce our framework as well as the training objective. In Sec. 3.3 we will describe how to train our model with an alternating optimization algorithm and an experience-replay buffer.
3.1 OPTIMIZATIOO-BASEE INFERENCE
Let F(x) = ý be a differentiable forward model that generates output ő given an input variable x E X. For example, ÿ might be an image, and x might be the latent variables for generative model. Given an observation y. the goal of OBI is to find the input X such that an objective function L(ğ. y) is minimized. Formally, we write this procedure as:
= argmin L(F()) xeX
When the objective function I. and the model F are both differentiable, we can perform the optimiza- tion over input space X with gradient descent.
3.2 REMAPPING THE INPUT SPACE
Instead of operating in the original input space X, we will create a new space where gradient descent is efficient and converges in small number of iterations. We will use neural network
3

---

Under review as a conference paper at ICLR 2023
Figure 2: Landscape Learn- ing. An optimization trajec- tory {z, Ja=o collected is used to train O that corresponds to a higher L, will yield higher gradient when training O Optimization over multi- ple steps along the trajectory causes θ to learn patterns of trajectories and create more efficient loss landscape.
Optimization Trajectory of
Gradient Direction of New Landscape
Original Landscape
Zt R2
2g
40
44
25
Collecting Trajectories in
Updating e
Z + that maps from the new space Z to the original space X. The parameters of the mapping C is the vector 0. The learning problem we need solve is to estimate 0 so that there is short gradient descent path in Z from the initialization to the solution. Fig. shows an overview of this setup.
We formulate objective by rolling out the gradient updates z, where we write Z₃ 24-1 AL -z as the h update with step size of X. For gradient descent in space Z to be efficient, the goal of our introduced θ is to move every step Z, as close as possible the global minima:
T Σ L(F(((ZZ))) v)
0, Zt-1
1 =
i argmin ((z.a)
where
OL
t
Oz.
We visualize this process with toy example Fig. Gradient updates on θ w.r.t multiple steps z, along a trajectory will cause the loss value on each step to be lowered. By learning the parameters of across many examples, θ can learn the patterns of optimization trajectories in X. For example, θ can learn to estimate the step size in X and dynamically adjust it to each example. Moreover, θ can learn to reduce non-convexity of the landscape.
Once we obtain with parameters ò, we do inference on new example y through the standard optimization-bant based inference procedure, except in Z space now. Given the observation y, we find the corresponding x through the optimization:
+ where = argmin L(F(((Z)).v)
262
When the inverse problem under-constrained, one can infer multiple hypotheses for x by repeating the above optimization multiple times with a different random initialization for Zg.
3.3 TRAINING
We use alternating optimization (AO) in order to train θ jointly with estimating z for each example in the training set. Specifically, first fix parameters of θ and collect N optimization trajectories of z, each with length T. Adopting the same terminology from the robotics community for learning on continuous states Mnih al. (2013), we term this an experience replay buffer. Subsequently, we randomly sample data from this buffer and train θ to optimize the loss function. We alternate between these two steps for a fixed number of iterations with gradient descent for both. Depending on the appli- cation, the training time for θ varied from one hour to one day using four GPU server. The complete algorithm is provided in algorithm Please see the appendix for more implementation details.
We also experimented with online version of the training algorithm, where update parameters of θ immediately after one update to z. However, in our experiments, we found this resulted in a slower convergence rate. We show these comparisons in the ablation experiments.
4 EXPERIMENTS
The goal of our experiments is to analyze how well our proposed method can be applied to various existing OBI methods to improve the optimization efficiency. We demonstrate application of our method three diverse OBI methods in computer vision, including both generative models and
4

---

Under review as a conference paper at ICLR 2023
Algorithm Learning Mapping Network +
1: Input: Ground truth y, step size À and As number of buffers B, number of data samples in
buffer N, number of optimization steps per sample T, loss function L, and forward model
2: Output: Learned mapping network è
3: Randomly initialize a mapping network & with parameters
4: for 1, ...., B do
5: Initialize Experience Replay Buffer (z4,)))=11
6: for 1,... N. do
7:
Zo +- 0
8: for 1,.,, do
Collecting optimiza- tion trajectory in Z while keeping 0 fixed
9: Z, + Zt-1 + A= dz. ((F(O(Z))
10:
Zt,i s 2t
end for
12:
end for
for 1;,,.., T N do
14:
Randomly sample from previously collected buffer
Updating 0 with previ- ously collected latents
15:
0j 4 0;-1 + Ao 20,-1 ((P(O())),
16:
end for
end for
18:
Return 0 =0
discriminative models. For each OBI method, the inference-time optimization objective of the baseline and ours can written as:
Baseline:
min L(F((x),)) xEX
Ours: 2 min L(F(((2)))
2EZ
Next, we provide the specific implementation of the loss term for each application, along with quantitative and qualitative results. We also perform experiments to understand the loss landscape in Sec. A and perform ablations on different parts of our approach in Sec. 4.4
4.1
GAN INVERSION
We first validate method StyleGAN inversion (Abdal et al., 2019). We take pretrained generator of StyleGAN Karras et al. (2019) denoted as F. Let be an observed image whose input variable are recovering, we optimize the objective of Eq. where the loss be written as,
L(j. v) Lipips (Ú. v)
|6
v11
where Lipips is a perceptual similarity loss introduced Zhang et al. 2018). 0 F(i) for baseline and ò = (2)) for ours. We train θ on the train split of CelebA- HQ Karras al. 2017) dataset and evaluate on CelebA-HQ validation split for in-distribution experiments and LSUN LSUN-CarYY Yu et al.) 2015) for distribution shifting (OOD) experiments. We compare the results from method against the statc-of-the encoder-based GAN inversion model (Richardson et al. 2021
Quantitative Results. From Fig. Bal we see that in all experiments, optimization in our space Z consistently outperforms the baseline from the first optimization step to after convergence. This gap in performance even larger when evaluated on OOD data. This suggests that the improvement in performance is not caused by memorizing the training data. Note that our image reconstruction performance after only 2 steps of optimization is able outperform or be on-par with the baseline at 20 steps of optimization, resulting in a 10-fold improvement in efficiency. Even after convergence (after 2000 optimization steps), our reconstruction performance improves over the baseline by 15% for in-distribution evaluation and 10% for OOD evaluation. When compared with encoder-based GAN inversion (Richardson et al. 2021). our method achieves better reconstruction after 11 steps of optimization for in-distribution data and 3 steps for OOD data.
Qualitative Results. From Fig. we can see that our method already shows improvements on in-distribution data it almost perfectly reconstruct details like fine hair strands, the cap on the person's head, the object in the person's 's mouth as well-as the text on it. Interestingly, our method is able to discover and reconstruct latents for cats while the encoder-based model fails miserably as
5

---

Under review as a conference paper at ICLR 2023
Ours
Ours (OOD)
0.10
Ours
Ours (OOD)
Baseline
Baseline (OOD)
Baseline
Baseline (OOD)
Encoder
Encoder (OOD)
0.08
Loss
0.06
0.02
0.00
10
10
10
10
10P
10'
10a
Optimization Steps
Optimization Steps
(a) GAN Inversion
(b) 3D Human Posc Reconstruction
Figure 3: Optimization Performance. We visualize the trends of optimization performance compared with the baseline. In GAN Inversion (Left), we evaluate all models on test splits of CelebA-HQ (Karras et al. 2017) and LSUN-cat (Yu et al. 2015) (OOD) with loss defined in Eq.
5 Since encoder-based inference doesn't involve optimization, we use flat line to represent it. We perform 2000 steps of gradient descent for all models except encoder- based models. In 3D Human Pose Reconstruction (Right), we evaluate all models on test splits of GRAB (Taheri et al., 2020) and PROX (Hassan et al. 2019) (OOD) with loss defined in Eq. O
We perform 200 steps of gradient descent for all models. For each step, we plot the average loss value of test splits.
shown in Fig The performance on OOD data truly highlights the benefits of our method. We also visualize how the face generations evolve over the process of optimization in Fig. 4 We can see that in just 4 steps, our method is already able to reconstruct coarse features such as face orientation, skin tone and hair color, while the baseline has hardly deviated from the initialization in regard to any of these features. Further, in Fig. [6 we visualize reconstructions from partial observations where only the center of the face (row 1) or everything other than the mouth (row 2) is visible. We can see variety of feasible possibilities for the hidden regions (c.g., different hairstyles, lip colors, expressions, etc) showcasing the diversity of the new latent space. More uncurated examples of image inpainting and outpainting can be found in Fig.
4.2
3D HUMAN POSE RECONSTRUCTION
In addition image generation, our framework also works for 3D reconstruction. For this, use VPoser (Pavlakos et al. 2019) a variational autoencoder (Kingma & Welling 2013) that has learnt pose prior over body pose. VPoser was trained on SMPL-X body pose parameters y g obtained by applying MoSh (Loper et al. 2014) on three publicly available human motion capture datasets: CMU Mocap CMU training set of Human3, 6M (Ionescu al. 2014), and the PoscPrior
dataset (Akhter& & Black |
We take a pretrained VPoser decoder denoted as F. Our trained mapping network projects a vector from the new input space z E Z to vector in the original VPoser decoder's 's input space x E X. Similar to GAN Inversion, we optimize the objective of Eq. where the loss function between predicted and ground truth pose parameters is,
L(ū. v) = 110 - y
(6)
where ý = F(t) for the baseline and ý F()(2)) for ours. For training +, we use the GRAB dataset (Taeri et al., 2020) which contains poses of humans interacting with everyday objects. We construct splits for novel video sequences - thus the test split will contain a seen human subject but potentially unseen pose / demonstration by that subject. We evaluate in-distribution experiments and on the PROX dataset (Hassan et al. 2019) for OOD experiments, which contains poses of humans interacting in 3D scenes (e.g., living room, office, etc).
this test split for
Quantitative Results. For SMPL-X human pose reconstruction experiment, the results follow a similar trend GAN inversion, with massive improvement in both convergence speed and final loss values after convergence (see Fig. (bb.. Our method outperforms the baseline by 19% for in-distribution evaluation and 1 1% for OOD evaluation.

---

Under review as a conference paper at ICLR 2023
IIage
Ours
Ground Truth
25
Step
Step
Step
Step 2000
Step
Step 5
Step 10
... Step 2000
Figure Optimization Process for GAN Inversion. Comparing optimization process of our method and the baseline in order to reconstruct the ground truth image. Left shows the results from the baseline where optimization is done the original input space X. Middle shows the results from our method where optimization is done in our space Z. Right column contains the ground truth image to each example. Each row corresponds the same example.
VPoser
Ours
Ground Truth
Step
Step 3 Step 6 ... Step 200
Step
Step 3
Step 6 ... Step 200
Figure 5: Optimization Process for 3D Human Pose Reconstruction. Results shown for out-of-distribution PROX dataset for sitting (Top) and standing (Bottom) poses.
Qualitative Results. In Fig. we visualize how the human pose reconstructions evolve over the process of optimization. Here, we observe that the reconstructions from steps 0 to 6 of the baseline are similar for both examples. On the other hand, our method caters to fast convergence for the varying examples, highlighting the general, yet efficient properties of our search space. Further, in Fig. we visualize reconstructions from partial observations where the only joints visible are that of the upper body (row 3) lower body (row 4). We obtain a wide range of feasible possibilities for the hidden joints demonstrating the diversity of the latent space.
4.3 DEFENDING ADVERSARIAL ATTACKS
Our method also applicable to discriminative models. state-of-the-a art defense (Ma et al, 2021) for adversarial attack optimizes the self- ceff-supervision task inference time, such that the model can dynamically restore the corrppted structure from the input image for robust inference. Following the existing algorithm implementation, optimize the input image via method by minimizing the
7

---

Under review as a conference paper at ICLR 2023
Partial GT
Reconstruction from Random Initializations
Figure 6: Diversity of Masked Reconstructions. We visualize reconstructions for partially observ- able inputs from random initializations. The masked regions i.c. the gradient is set to be zero. By optimizing only on the partial observation, we obtain diverse, feasible solutions for the hidden regions. or 4 out of 20 most representative samples are presented.
considered for loss computation,
following discriminative loss function:
log exp(cos(f;, ff) En exp(cos f, fx)) + Ol1O())11 (7)
L(F(r+ L a). Ex,
(8) Yij
where a is the adversarial attacks that aim defend against, O(z) is our additive defense vector to optimize, f; are the contrastive features produced by the neural network F from the 7th (s) image instance, and y is the indicator for the positive pairs and negative pairs.
After obtaining the mapping network θ and the input variable 2, the prediction is ý (a O())), where F' the classification model. Note that the a self-supervision loss is optimized as a proxy for increasing the robust classification accuracy. In addition, we add L-g norm decay term for the generated noise = to avoid generating reversal vector that is too large.
Optimization Steps
No Optimization
step Baseline
steps Baseline
steps
Model Type
Ours
Ours
Baseline
Ours
RO
31.99
34.62
44.65
36.77
44.23
38.38
43.43
AWP
35.61
39.54
51.39
42.81
51.67
44.96
51.05
MART
35.66
39.77
51.77
42.50
51.77
45.42
50.96
SemiSI.
29.78
34.53
52.11
37.27
51.23
40.93
49.83
Table 1: Experiment improving adversarial robust accuracy. Our goal is to defend 200 steps of La = 256 225 norm bounded attack (Madry et al., 2017), where the attack's step size 64/255. Our baseline is the SOTA test-time optimization-baan based detense (Mao et al. 202 which minimizes the loss self-supervision task.
Quantitative Results. We evaluate our method on four popular pretrained robust models (Rice)
et al. 2020; Wang et al. 2020: Wu et al. 2020: Carmon et al. 2019) on CIFAR-10 dataset. The results in Mao et al. (2021 require many steps to optimize the objective to improve the adversarial robustness, which slows down the inference by hundreds of times than the original forward pass.
Ideally, desire test-time optimization that can adapt to the attacked images in just one step, causing
8

---

Under review as a conference paper at ICLR 2023
In-Distribution
Out-of-Distrbuttio
Encoder
Ours
Ground Truth
Encoder
Ours
Ground Truth
Figure 7: Comparison Against Encoder-Basee Inference. Left shows the results on the test split of CelebA-HQ: Right shows the results on the LSUN-cat dataset.
Number of Steps
Full Model
Without AO and Buffer
Random θ
Baseline
In-distribution
20
3.082
3.583
4.417
4.456
OOD
20
4.932
5.127
5.135
5.292
In-distribution
200
1.964
3,034
2.723
2.569
OOD
200
3.498
4.756
3.823
4.617
Table 2: Ablation Study on mapping network. Number of Steps indicates the number of optimiza- tion steps performed during inference.
the minimal delay at inference time. In Table I our method outperforms the gradient descent method Mao et al. (202T) by up to 18% robust accuracy using a single step, providing a real-time robust inference method. Note that our method converges after step of optimization, demonstrating the effectiveness of our approach.
4.4 ABLATION STUDY
In this section, we present an ablation study by removing the proposed alternating optimization (AO) scheme and the experience replay buffer. We also compare against θ that is randomly initialized. From Table in appendix, we discovered that for in-distribution, AO and training of Theta improves the optimziation performance 14% and 30% respectively. Such gap becomes 35% and 28% for evaluation on 200 steps. For OOD data, the advantage is furthered enlarged as shown in Table
One surprising result discovered experimentally is that OBI under randomly initialized mapping network @ consistently outperforms the baseline. We believe this is due to the fact that adding Gaussian distribution an underlying latent distribution StyleGAN is beneficial in smoothing out loss landscape, making it easier to perform OBI. Similar random projection can also be found in Wu ct al. 2019).
CONCLUSION
This paper presents a method accelerate optimization-baser inference to invert forward model. We propose an approach that learns new space that is easier than the original input space to optimize with gradient descent at testing time. Our experiments and analysis on three different applications in computer vision have shown that by learning this mapping function, optimization becomes more efficient and generalizes better to out-of-distribution data. Through quantitative and qualitative analysis, we found that such improvement in optimization performance comes from more efficient loss landscape. Since optimization-bans inference has many advantages over encoder-base based inference, believe methods to accelerate them will have many impacts in variety of applications.

---

Under review as a conference paper at ICLR 2023
6 ETHICS STATEMENT
Optimization-basee inference has wide variety of applications broadly in computer vision, robotics, natural language processing, assistive technology, security, and healthcare. Since our proposed method provides sigmificant acceleration these inference techniques, we expect our work to find positive impact in these applications, where speed, accuracy, and robustness are often critical.
Our algorithm compatible with many different types of forward models - as long as a gradient can be calculated - including neural networks. However, learned forward models are known to acquire and contain biases from the original dataset. Our approach will also inherit the same biases. While our approach to inference offers some advantages to out-of-distribution data, it does not mitigate nor correct biases. The datasets in our experiments are likely not representative of the population, and consequently also contain biases. We acknowledge these limitations, and applications of this method should be mindful of these limitations, especially in potentially sensitive scenarios. As the community continues address biases in models and datasets, our method can be applied to these models in the future too.
As authors of the paper, have carefully reviewed and will adhere to the code of ethics provided at https://iclr. http://cw....o/pbbbc////oeoet///
7
REPRODUCIBILITY STATEMENT
We will release all code, models, and data. Here we provide the implementation details.
7.1 GAN INVERSION
Mapping network θ is implemented with a 3-layer MLP. The input dimension (dimension of Z space) is the same as the output dimension (dimension of X space). For each intermediate output, we apply a Leaky ReLu function with negative slope of 0.2 as an activation function. Hidden dimension of the MLP is 1024. For optimizing (collecting optimization trajectories), we use an Adam optimizer with learning rate of 0.1. For training the mapping network O, we use an AdamW optimizer with a weight decay of 0.1 with learning rate of 0.0001. We used the following parameter set, T = 20, N 256, = 500. For baseline, we use the implementation of Abdal et al. (2019) provided (hee The pretrained weights of StyleGAN converted to Py Torch are also provided in the same link.
7.2 3D HUMAN POSE RECONSTRUCTION
Mapping network θ is implemented with 3-layer MLP. The input dimension is 128 (dimension of Z) and the output dimension 32 (dimension of X). For each intermediate output, we apply a Leaky ReLu function with negative slope of 0.2 as activation function. Hidden dimension of the MI.P is 512. For optimizing = (collecting optimization trajectories), we use an Adam optimizer with learning rate of 0. For training the mapping network θ, we use AdamW optimizer with a weight decay of 0.1 and a learning rate of 0.005. We use the following parameter set, T = 200, N 40960, B 500. For fair comparison with the baseline, we tried a range of learning rate for x E X {0.5, 0.1, 0.05, 0.01, 0.001} and select the best performing configuration for comparison.
7.3 DEFENDING ADVERSARIAL ATTACKS
Mapping network θ is implemented with 3-layer MLP. The input dimension is 3072 (dimension of Z) and the output dimension is 3072 (dimension of X). For each intermediate output, we apply a Leaky ReLu function with a negative slope of 0.2 as an activation function. Hidden dimension of the MLP is 3072. For optimizing z (collecting optimization trajectories), we use an Adam optimizer with learning rate 0.2/250 For training the mapping network O we use AdamW optimizer with learning rate of 0.0001 and a weight decay of 0.1. We use the following parameter set, T = 5. N 5120, B = 70. For the regularization term À that constrains the amplitude of the additive defense vector, we use À = 1. We use random start instead of zero start for initializing the attack reversal vector.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4432-4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Ijaz Akhter and Michael Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. 06 2015. doi: 10. (0.1109/CVP2.221552297773
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. International Conference on Machine Learning, pp. 146-155. PMLR, 2017.
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learnnng to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
David Bau, Jun- Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what a gan cannot generate. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4502-455 1, 2019.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using generative models. In International Conference Machine Learning, pp. 537-546. PMLR, 2017.
Philémon Brakel, Dirk Stroobandt, and Benjamin Schrauwen. Training energy-based models for time-series imputation. The Journal of Machine Learning Research, 14(1):2771-7977, 2013.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, pp. 39-57, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang, Unlabeled data improves adversarial robustness. H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https:::p//.o...... http:pppp///ooop..iiii htwp:/ewwwee.ino.......eeeeeeeee neuripssssss cc/paperr/019 htt.g/g......rssss.aa/e/e/222222
file File/32e0ba1 2eee33eeeeelo1g3eeeaaa2222222222 32
www.l15ddd-aappp..ee Paper. pdf
Boyuan Chen, Robert Kwiatkowski, Carl Vondrick, and Hod Lipson. Full-body visual self-modeling
of robot morphologies. arXiv preprint arXiv:2111.0633 2021a.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and benchmark. arXiv preprint arXiv:2103.12822
2021b.
Yunjey Choi, Youngiung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8188 1898,119, 2020.
CMU. Cmu mocap dataset.
Jochen L Cremer, Ioannis Konstantelos, and Goran Strbac. From optimization-basee machine learning to interpretable security rules for operation. IEEE Transactions on Power Systems, 34(5): 3826-3836, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- free attacks. In ICML, 2020.
Katherine Crowson, Stella Rose Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vlgan-clip: Open domain image generation and editing with natural language guidance. ArXiy, abs/2204.08583, 2022.

---

Under review as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Tan Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetwork. arXiv preprini arXiv:2/ 2021.
Justin Domke. Generic methods for optimization- modeling. In Artificial Intelligence and Statistics, pp. 318-326. PMLR, 2012.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Advances in neural information processing systems, 31, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning, In International Conference on Machine Learning, pp. 1920-1930. PMLR, 2019.
Mario Geiger, Leonardo Petrini, and Matthicu Wyart. Landscape and training regimes in deep
learning. Physics Reports, 924:1-18, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. Proceedings of the IEEE/CVF international conference on computer vision, pp. 2282-2292. 2019.
Carlos Hernandez, George Vogiatzis, and Roberto Cipolla. Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(3):548-554, 2008
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey. IEEE transactions on pattern analysis and machine intelligence, 44(9): 5149-5169, 2021.
Minyoung Huh, Richard Zhang, Jun- Jun-Yan Zhu, Sylvain Paris, and Aaron Hertzmann. Transforming and projecting images into class-conditions generative networks. In European Conference on Computer Vision, pp. 17-34. Springer, 2020.
Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6.m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325-1339 2014. doi: 10.1109/TPAM.20013 248.
Ali Jahanian, Lucy Chai, and Phillip Isola, On the" steerability" of generative adversarial networks. arXiv preprint arXiv: /907_0707111 000000077/, 2019.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: A review. IEEE transactions visualization and computer graphics, 26(11):335--3385 2019.
Kyoungkook Kang, Seongtae Kim, and Sunghyun Cho. Gan inversion for oft-of-range images with geometric transformations. Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 13941-13949, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEÉ/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
12

---

Under review as a conference paper at ICLR 2023
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Kyoung Mu Lee and C-CJ Kuo. Shape from shading with a linear triangular element surface model. IEEE Transactions Pattern Analysis and Machine Intelligence, 15(8):815-88 822, 1993.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiet prompt tuning. ArXiv, abs/2104.08691, 2021.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Haoran Su, and Qiang Liu. Fuse- dream: Training-fie free text-to-image generation with improved clip+gan space optimization. ArXiv, abs/2112.01173. 2021.
Matthew Loper, Naureen Mahmood, and Michael J. Black. Mosh: Motion and shape capture from sparse markers. ACM Trans. Graph., 33(6), nov 2014. ISSN 0730-0301, doi: 10.1145/2661229. 2661273. URL https:://d/i org/ /0. 11 229 2661273
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM transactions graphics (TOG), 34(6):1-16, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1rXiv:1706.06833 2017.
Chengzhi Mao, Ziyuan Zhong, Junfeng Yang, Carl Vondrick, and Baishakhi Ray. Metric learning for adversarial robustness. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Chengzhi Mao, Mia Chiquier, Hao Wang, Junfeng Yang. and Carl Vondrick. Adversarial attacks are reversible with natural supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 661-671, 2021.
David McAllester. Expectation maximization lecture notes of statistical methods for artificial
em. em. pdf
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 2437-2445, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv:1312.5602, 2013.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803 18000090.000999 2018.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Guim Peramna,, Joost Van De Weijer, Bogdan Raducanu, and Jose M. Álvarez. Invertible conditional gans for image editing. arXiv preprint arXiv: 16/ 1. 1.06355, 2016.
13

---

Under review as a conference paper at ICLR 2023
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and lly Sutskever. Learning transferable visual models from natural language supervision. In ICMI, 2021.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning, 2020.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. P Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287-2296, 2021.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
Saeed Soori, Bugra Can, Baourun Mu, Mert Gürbüzbalaban, and Maryam Mehri Dehnavi. Ten- grad: Time-efficient natural gradient descent with exact fisher-block inversion. arXiv preprint (:Xiv:2106.03947, 2021.
Veselin Stoyanov, Alexander Ropson, and Jason Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 725-733.
JMLR Workshop and Conference Proceedings, 2011.
Christian Szegedy, Wojeiech Zaremba. Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks.
2013.
Omid Taheri, Nima Ghorbani, Michael J Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In European conference on computer vision, pp. 581-600. Springer, 2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Transactions Graphics (TOG), 40(4):1-14, 2021.
Austin Tripp. Erik Daxberger, and José Miguel Hernández-Lobato, Sample-efficient optimization in the latent space of deep generative models via weighted retraining, Advances in Neural Information Processing Systems, 33: 1259-1192221 1122, 2020.
Hao Wang, Chengzhi Mao, Hao He, Mingmin Zhao, Tommi S Jaakkola, and Dina Katabi. Bidirec- tional inference networks: A class of deep bayesian networks for health profiling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 766-773, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Weiming Zhang, Lu Yuan, Gang Hua, and Nenghai Yu. A simple baseline for stylegan inversion. ArXiv, hawss//1040066.. 2021.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688. Citeseer, 2011.
Olga Wichrowska, Niru Maheswaranathan. Matthew W Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein Learned optimizers that scale and generalize. In International Conference Machine Learning, pp. 3751-3760. PMLR, 2017.
14

---

Under review as a conference paper at ICLR 2023
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. In NeurIPS, 2020.
Yan Wu, Mihaela Rosca, and Timothy Lillicrap. Deep compressed sensing, In International Conference on Machine Learning, pp. 6850-6860. PMLR, 2019.
Ziqi Yang, Jiyi Zhang, Ee-Chien Chang, and Zhenkai Liang. Neural network inversion in adversarial setting via background knowledge alignment. In Proceedings of the 20/9 ACM SIGSAC Conference on Computer and Communications Security, pp. 225-240, 2019.
Alex Yu, Sara Fridovich- Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:21
2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.00 03366, 2015.
Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1768-1777 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 586-595, 2018.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, pp. 592-608. Springer. 2020.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natural image manifold. In European conference on computer vision, pp. 597-613. Springer, 2016.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX
A LOSS LANDSCAPE VISUALIZATION
To understand the underlying cause of the significant improvement in optimization brought by our mapping network 0, we visualize in Fig. g the loss landscape for performing optimization in the original input space X, and our projected space Z. To generate this visualization, we first perform 20 steps of optimization on the validation dataset collect set of recovered latents. We then perform principle component analysis (PCA) on these recovered latents to obtain two principle directions. Finally, for individual examples, we evaluate the loss for vertices on meshgrid spanned by the two principle directions where the center is the last step of optimization.
From the visualization, we can see that the loss landscapes of the baseline are highly non-convex and contain points whose loss significantly higher than its neighboring regions, while our loss landscapes are significantly smoother, with the "spikes" removed. Besides, our loss landscapes also tend to be steeper than the baseline ones. These two phenomena directly cause our method to perform gradient descent faster and stabler.
-1--0-3--
Figure 8: Visualizing Loss Landscape (Uncurated). Visualizing the loss landscape of StyleGAN inversion spanned by two principle directions. Top row shows examples of the loss landscapes corresponding to our space Z. Bottom row shows the loss landscapes corresponding the original input space X for the same 4 examples. Note that the top row and the bottom row have different minimum loss values on the landscape because the minimas are obtained by performing independent optimization runs in space Z and X respectively. Given a fixed number of optimization step, optimization in Z reaches lower loss values than optimization in X.
B EQUIVALENCE TO EXPECTATION-
We show that algorithm in is equivalent hard SespecattionMMxxiizzztnn (McAllester). First we rewrite the objective Eq. as:
i argmaax max
Po(z)
(8)
To optimize objective Eq. we perform the following algorithm:
Initialize 0 with Gaussian distribution
2. Repeat the following until 11: In Pa (O()))))(( converges:
(a) Expectation: pe(2,) = 8(2 2 = 2,), where 2 = argmax,, (o(2.)F((())))
(b) Maximization: 0 argmax, Ez-p [n (o (((()))
where 8 denotes the parameters of the mapping network e, and 8(.) denotes a Dirac delta distri- bution. Since both expectation and maximization steps strictly increase maintain the value of Po(y: ((O(z))) under gradient ascent, and the function bounded, the algorithm is guaranteed to converge.
16

---

Under review as a conference paper at ICLR 2023
C DIMENSION OF Z
As our method learns a mapping network ④ that maps vector from new latent space Z to vector in the original input space X. The dimension of Z becomes a hyperparameter. table 3 we perform ablation studies on the effect of the dimension of Z space on the performance of the learned mapping network. From table B we can see a consistent improvement on performance as we increase the number of dimension of Z. However, such improvement hits diminishing return when the dimension in the same order of magnitudes as the dimension X.
Dimension of Z 16
32
64
128
256
512
2048
Baseline
In-distribution
3.157
2.797
2.489
2.489
2.212
1.964
1.920
2.569
OOD
4.872
4.592
4.277
3.916
3.618
3.498
3.392
4.617
Table 3: Ablation studies on the dimension of Z. We trained variations of full model with different number of dimension of Z space, varying from 16 to 2048 given the dimension of input space X is 512. Numbers correspond loss values defined in Eq. 5
D MORE EVALUATION ON OUT-OF-DISTRIBUTIOO GENERALIZATION
Since our proposed mapping network θ is parameterized by a neural network, there's no guarantee that the learned mapping function is surjective. Therefore, we empirically study the generalization performance by testing mapping network trained on CelebA-HQ against a spectrum of datasets from very similar ones (in-distribution) to completely different ones (OOD).
D.1 SYNTHETIC SPECTRUM
We first created a synthetic version of this spectrum of datasets by varying the level of Gaussian noise injected into the images of CelebA-HQ. With a higher level of Gaussian noise injected into the original images, more of the original image content is corrupted, creating a distribution of images further away from the original test images.
level of Corruption NV00 0.0%)
W.0,0.13)
O.0,0.22)
(o0,0.33
(.0,0.42)
Improvement
29.53%
18.95%
12.57%
6.30%
6.72%
Table 4: Evaluation of mapping network trained with Celeba-HQ trainset and tested on the Celeba- HQ testset corrupted with different levels of Gaussian noise. The improvement calculated by the percentage improvement of loss defined by Eq. 5 from baseline (no mapping network) to ours (with mapping network) after 200 steps of optimization.
D.2 NATURAL IMAGE SPECTRUM
We then created a natural image version of this spectrum of datasets containing: CelebA-HQ (original in-distribution testset), AFHQ-Cat (center-aligned cat faces), L.SUN-Ca (unaligned cat images), Container-Ship ("Container Ship" class from ImageNet), and Gaussian noise with individual pixel sampled from (0.5, 0.5²). From left to right, images in the datasets vary from very similar to human faces to very differrntt
From results evaluated on both synthetic spectrum and real spectrum, we observed consistent improvements of our method over the baseline, which shows the generalization performance of method when applied to OOD data. From table 4 and 5 we see the improvements drop the evaluation data is less and less in in-distribution with the training data, which indicates that the mapping network does learn prior from the training data.
17

---

Under review as a conference paper at ICLR 2023
dataset
CelebA-HQ
AFHQ-Cat
LSUN-Cat
Container- Ship
(00.5 0.59)
Improvement 29.53%
35.99%
24.36%
16.87%
6.60%
Table 5: Evaluation of a mapping network trained with Celeba-HQ trainset and tested on a spectrum of datasets from the original CelebA-HQ (in-distribution) to Gaussian noise (OOD). AFHQ-Cat Choi et al. (2020) is a dataset with aligned cat faces. LSUN-Can Yu et al. (2013) is dataset with unaligned cat images. Container-Shee Ship is 200 images sampled from "Container Ship" class of ImageNet
Deng Deng
ct al. (2099) The improvement is calculated the same way as table
E DEPENDENCE ON TRAINING DATASET
From previous section, we know that the learned mapping network contains priors learned from the training data. To exclude such influence, we train our mapping network using randomly generated images made of Gaussian noise and evaluate on testset of CelebA-HQ. From table [ we see that even trained with the task of reconstructing images of Gaussian noise, mapping network still bring some improvement over the baseline model, though much less significant. Interestingly, this improvement number is consistent with the last columnes of table 4 and where mapping network is trained with CelebA-HQ and tested on images with Gaussian noise.
Baseline
Ours (trained on Gaussian noise)
Improvement
CelebA-HQ 2.569
2.396
6.72%
LSUN-Cat 4.617
3.717
19.49%
Table 6: Evaluation of a mapping network trained with images of Gaussian noise sampled from (0.55, 0.5²) clipped to [o, 1]. Numbers show the average loss defined in Eq. evaluuaed on testset of CelebA-HQ and LSUN-Cat.
F LIMITATIONS
Optimization-base inference has intrinsic advantages robustness, accuracy, and flexibility, which comes at the cost of additional computation time during inference. Encoder-based methods will usually perform faster because they only require single forward pass of neural network, while approach requires several computational passes in both the forward and backward (gradient) direction. We believe that for many applications this trade-off will be desirable, especially in cases where accuracy is more important than speed. Our approach aims to minimize this additional computational overhead brought by optimization based inference, and our experiments on multiple datasets show the significant computational savings compared to other optimization- based inference methods.
Unlike many other optimization- based inference algorithms, our approach also requires training step in order to fit suitable landscape, which requires both training time and training data. However, we believe this overhead insignificant for most applications and we have designed our neural networks to be efficient. For example, θ is relatively lightweight, making its training time fairly marginal compared to the training of the forward model F. In our experiments, we found that training time of O is orders of magnitudes faster than the training time for F.
18

---

Under review as a conference paper at ICLR 2023
Partial GT
Reconstruction from Random Initializations
Figure 9: Diversity of Masked Reconstructions. (Uncurated)) We visualize reconstructions for partially observable inputs from random initialization. The masked regions are not considered for loss computation, i.e., the gradient to be zero. By optimizing only the partial observation, we obtain diverse, feasible solutions for the hidden regions. All reconstruction results presented are randomly sampled.
19


Reviews:
[{"review_id": "rWKnOknRH80", "reviewer": "Reviewer_4P2g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework to accelerate and stabilize the process of neural network inversion. Specifically, the proposed method uses a neural network to learn a mapping between the latent spaces that allows gradient descent over the input to converge in fewer steps. Experiments show that the proposed framework can successfully improve the convergence speed for optimization.", "summary_of_the_review": "I think this is an interesting paper to accelerate neural network inversion. Although the proposed method is rather simple, it seems to be effective. I am willing to raise my recommendation if the question mentioned in the weakness section is addressed.\n", "strength_and_weaknesses": "### Strength\n* The paper is well-written and easy to follow. \n* The motivation is clear, and the intuition behind the proposed method is straightforward.\n* The experiment results seem promising. Inverting a neural network has a lot of advantages and downstream applications. The proposed * method appears effective and can further accelerate and stabilize such a process.\n### Weaknesses\nMy primary concern is regarding the sub-optimal solution when using the proposed method. Is there any circumstance of Z that may result in a suboptimally averaged solution since the loss landscape is a smoothed-out version of X? If so, it is reasonable, as it can be considered a tradeoff between speed and quality. However, showcasing such examples and quantitatively measuring the tradeoffs would be better.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NvYXuupAX10", "reviewer": "Reviewer_PZA2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores optimization-based neural network inversion. Instead of optimizing in the original input space, they propose to learn a new latent space and run optimization in the learned space during test time. They show that their technique greatly accelerates the inversion optimization process for a variety of tasks including GAN inversion, adversarial defense, and human pose reconstruction.", "summary_of_the_review": "I recommend weak acceptance of this paper. On the one hand, this paper proposes a very simple technique that solves an important problem and they have applied their approach to a sufficient number of problems. On the other hand, I don't think I fully understand how their approach works and I think there should be more study (either theoretical justification or empirical study as suggested in the \"weakness\" section).", "strength_and_weaknesses": "**Strength:**\n1. The proposed technique is very simple yet it has been shown to be effective (greatly speeding up the inversion process) on several tasks involving neural network inversion.\n2. The writing is easy to follow. The illustrations are helpful for understanding. The mathematical and experimental details are clear to me.\n\n**Weaknesses:**\nMore empirical studies need to be done in order to support their claim and fully understand the proposed technique:\n1. Firstly, does the learned latent space only depend on the model architecture, not the data? To what extent can we generalize out of domain? Can it be used to accelerate inference using the same model but with a *very* different dataset? I understand that the authors have already conducted experiments in the OOD scenario, but it would convince me even more if we can see the whole spectrum going from very similar datasets to completely different datasets (This analysis can be done using even synthetic data in a controlled setting). In addition, I would even suggest using random inputs to generate data and learn the latent space and run inferences using real data. It would be very interesting if we could still observe any improvement at all.\n2. Secondly, it would be better if we could understand how much the learned latent space depends on the model architecture. For the purpose of analysis, what if we use the same dataset and a different model architecture? \n\nI am not fully convinced by their statements regarding:\n1. They mentioned that OBI supports generating diverse hypotheses while EBI can only generate one outcome. I think this might not be true as you can always inject noise into the encoder to generate diverse samples. One can also parameterize their encoders to output distributions over inputs.\n2. They stated in the second paragraph that OBI can adapt without additional training. Will that still be true with the proposed learned new latent space or will the performance drop (at least slightly) with partially missing observations or new data distributions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "L-yiDrzaYT", "reviewer": "Reviewer_A9dv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method that learns a loss landscape\nwhere gradient descent is efficient, bringing massive improvement and acceleration\nto the inversion process of learning a CNN. Advantages are demonstrated using a number of methods\nfor both generative and discriminative tasks.\n\nThe method helps to accelerate optimization-based inference to invert a forward model. The concept involves\nlearning a new space that is easier than the original input space to optimize with gradient descent at testing time.\n", "summary_of_the_review": "\n\nInversion of FF models will help solving many inverse problems.\nThe proposed method may help researchers to explore more work in this direction.\n\nThe lack of sufficient analytics is a major cause of worry.\n \n\n", "strength_and_weaknesses": "\nPaper is easy to follow and read. Enough of empirical results provided to substantiate the claim.\n\n\nAlgorithm 1 is not referred in text; although trivial matter - bit it should be done at least once.\n\"MAML\" is not specified anywhere in document- pl, expand the term.\n\nIs your method similar to applying ADMM over a manifold/topological space, to optimize ?\n\nWould have preferred more analytics concerning the convergence of your algorithm  - does that have any minimal guarantee ?\n\nYou talk of patterns of trajectories being learned from latent space.\n How many such pattern clusters exist? DO they sufficiently represent categories of populations required\nwith no bias?\n\n\nFew references not cited, some example:\nLandscape and training regimes in deep learning; Mario Geiger, Leonardo Petrini, Matthieu Wyart; Physics Reports, Volume 924, 15 August 2021, Pages 1-18.\n\nNeural Network Inversion in Adversarial Setting via Background Knowledge Alignment; Ziqi Yang et al; Proceedings CCS '19; Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment; https://dl.acm.org/doi/abs/10.1145/3319535.3354261\n\nT. Hospedales, A. Antoniou, P. Micaelli and A. Storkey, \"Meta-Learning in Neural Networks: A Survey\" in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 44, no. 09, pp. 5149-5169, 2022.\ndoi: 10.1109/TPAMI.2021.3079209\n\nTENGRAD: TIME-EFFICIENT NATURAL GRADIENT DESCENT WITH EXACT FISHER-BLOCK INVERSION; Saeed Soori et. al; \nhttps://www.cs.toronto.edu/~mmehride/papers_private/TENGRAD.pdf\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "75svTyFnE8A", "reviewer": "Reviewer_AFoq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the problem of neural network inversion, which lies at the core of many downstream tasks incorporating neural networks, such as inverse problems. Naively inverting a network via gradient descent is challenging, due to the induced non-convex landscape with potentially many local minima. The authors present a method that takes a pretrained neural network $F$ and learns an encoder $\\Theta$ such that gradient descent is easier to perform over the input space of $F \\circ \\Theta$, as opposed to directly over the input space of $F$. The model is trained by minimizing a loss over many trajectories of gradient descent for a fixed number of steps. The authors apply their approach to GAN inversion, 3D pose reconstruction, and protecting against adversarial attacks. The proposed approach shows that gradient descent empirically requires fewer steps to approach a solution as compared to direct inversion.", "summary_of_the_review": "Overall, I found the ideas in the paper intuitive and the results look fairly compelling. ", "strength_and_weaknesses": "**Strengths:**\n- The proposed training strategy allows for gradient descent to be performed more quickly in the latent space in that reasonable solutions are found with fewer iterations.\n- The new encoder performs better on out of distribution images for inversion than standard encoder-based inference approaches.\n\n**Weaknesses:**\n- The way the methodology is proposed, the encoder would be specific for the forward model considered, rather than simply to the GAN or generator being inverted. I wonder if the authors could comment on a scenario that could help broaden the methods scope (in the next section).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject