Gradient Origin Networks
Authors: Sam Bond-Taylor
Chris G. Willcocks
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Learning
Generative Models
Implicit Representation

Abstract: This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points. The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.

Published as a conference paper at ICLR 2021
GRADIENT ORIGIN NETWORKS
Sam Bond-Tay Wand-Taylor' & Chris G. Willcocks*
Department of Computer Science
Durham University (samuel. e e.oond-oo- boond-tayor,, whristopher..gg.. www.looor...i..oookkk.kk.o..o.o. @durham. ac.
ABSTRACT
This paper proposes new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which implemented by initialising a latent vector with zeros, then using the gradient of the log- se-likelihood of the data with respect this zero vector as new latent points, The approach has similar characteristics to autoencoders, but with simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.
INTRODUCTION
Observable data in nature has some parameters which are known, such as local coordinates, but also some unknown parameters such as how the data is related to other examples. Generative models, which learn distribution over observables, are central our understanding of patterns in nature and allow for efficient query of new unseen examples. Recently, deep generative models have received features when modelling data distributions. As
interest due their ability to capture a broad such, they offer direct applications such as synthesising high fidelity images (Karras et al., 2020), super-resolution (Dai et al., 2019), speech synthesis (Li et al., 2019), and drug discovery (Segler et al., 2018), 8), as well as benefits for downstream tasks like semi-supervised learning (Chen et al., 2020).
number of methods have been proposed such as Variational Autoencoders (VAEs, Figure la), which learn to encode the data to latent space that follows normal distribution permitting sampling (Kingma & Welling, 2014). Generative Adversarial Networks (GANs) have two competing networks, one which generates data and another which discriminates from implausible results (Goodfellow et al., 2014). Variational approaches that approximate the posterior using gradient descent (Lipton & Tripathi, 2017) and short run MCMC (Nijkamp et al., 2020) respectively have been proposed, but to obtain a latent vector for sample, they require iterative gradient updates. Autoregressive Models (Van Den Oord et al., 2016) decompose the data distribution the product of conditional distributions and Normalizing Flows (Rezende & Mohamed, 2015) chain together invertible functions; both methods allow exact likelihood inference. Energy-Basee Models (EBMs) map data points to
D
x
&
F
F
8
X
h Oe
(a) VAE
(b) GON
(c) Implicit GON
(d) Variational GON
Figure I: Gradient Origin Networks (GONs; b) gradients (dashed lines) encodings thus only single network F is required, which can be an implicit representation network (c). Unlike VAEs (a) which networks, E and D. variational GONs (d) permit sampling with only network.
Authorr contributed equally.

---

Published as a conference paper at ICLR 2021
energy values proportional to likelihood thereby permitting sampling through the use of Monte Carlo Markov Chains (Du & Mordatch, 2019). In general to support encoding, these approaches require separate encoding networks, are limited to invertible functions, require multiple sampling steps.
Implicit representation learning (Park et al., 2019; Tancik et al., 2020), where a network is trained on data parameterised continuously rather than in discrete grid form, has seen surge of interest due to the small number of parameters, speed of convergence, and ability to model fine details. In particular, sinusoidal representation networks (SIRENs) (Sitzmann al., 2020b) achieve impressive results, modelling many signals with high precision, thanks to their use of periodic activations paired with carefully initialised MLPs. So far, however, these models have been limited to modelling single data samples, or use an additional hypernetwokk or meta learning (Sitzmann et al., 2020a) to estimate the weights of simple implicit model, adding significant complexity.
This paper proposes Gradient Origin Networks (GONs), a new type of generative model (Figure 1b) that do not require encoders or hypernetworks. This is achieved by initialising latent points the origin, then using the gradient of the log-likelihood of the data with respect to these points as the latent space, At inference, latent vectors can be obtained a single step without requiring iteration. GONs are shown to have similar characteristics to convolutional autoencoders and variational autoencoders using approximately half the parameters, and can be applied to implicit representation networks (such as SIRENs) allowing space of implicit functions to be learned with simpler overall architecture.
2 PRELIMINARIES
We first introduce some background context that will be used to derive our proposed approach.
2.1 EMPIRICAL BAYES
The concept of empirical Bayes (Robbins, 1956; Saremi & Hyvarinen, 2019), for a random variable z ~ Pa and particular observation Zu Pzu. provides an estimator of expressed purely in terms of p(zo) that minimises the expected squared error. This estimator can be written as conditional mean:
zp(z)z (o)z)zo)dz = p(z, Zo)
2(Z0)
dz.
(1)
p(zo)
Of particular relevance is the case where Zo is noisy observation of z with covariance E. In this case p(zo) can be represented by marginalising out z:
p(zo)= = (2 (2r)4/2| det exp (- (zo (-(zo-
2) Σ (zo 10012//(2)())))) p(z)dz.
Differentiating this with respect to 2n and multiplying both sides by E gives:
VV.)(()) = f
p(z, zo 2o)dz
zp(zz (z, zo) 20)dz - zop(Zo)
(3)
After dividing through by p(za) and combining with Equation we obtain a closed form estimator of z (Miyasawa, 1961) written in terms of the score function V log p(zo) (Hyvärinen, 2005):
2 (Zo) = Zo + EV., p(zo)
(4)
This optimal procedure is achieved in what can be interpreted as single gradient descent step, with no knowledge of the prior p(z). By rearranging Equation 4, a definition of V log p(zo) can be derived; this can be used to train models that approximate the score function (Song & Ermon, 2019).
2.2 VARIATIONAL AUTOENCODERS
Variational Autoencoders (VAEs; Kingma & Welling 2014) are a probabilistic take on standard autoencoders that permit sampling. A latent-based generative model po (x z)) is defined with normally distributed prior over the latent variables, Po = [z 0, Ia). po (x zz) is typically parameterised as a Bernoulli, Gaussian, multinomial distribution, or mixture of logits. In this case, the
true posterior po po(x|x (z|x) is intractable, so a secondary encoding network yo x) is used to approximate the true posterior; the pair of networks thus resembles a traditional autoencoder. This allows VAES to approximate pe (x) by maximising the evidence lower bound (ELBO), defined as:
log (x) > VAE DxL. (N (N (z) (o) Ia)) (x (lo (x|z))
(5)
2

---

Published as a conference paper at ICLR 2021
To optimise this lower bound with respect to 0 and Φ. gradients must be backpropagated through the stochastic process of generating samples from ~ (z(x). This is permitted by reparameterising z using the differentiable function z' ju(z) + (z) ) O 6, where e ~ (O, Ia) and (u(z) and a (z)² are the mean and variance respectively of multivariate Gaussian distribution with diagonal covariance.
METHOD
Consider some dataset x ~ Pa of continuous or discrete signals x IR",, it typical to assume that the data can be represented by low dimensional latent variables z E R which can be used by a generative neural network to reconstruct the data. These variables are often estimated through the use of a secondary encoding network that is trained concurrently with the generative network. An encoding network adds additional complexity (and parameters) to the model, it can be difficult to balance capacities of the two networks, and for complex hierarchical generative models designing suitable architecture can be difficult. This has led some to instead approximate latent variables by performing gradient descent on the generative network (Bojanowski et al., 2018; Nijkamp et al., 2020). While this addresses the aforementioned problems, it significantly increases the run time of the inference process, introduces additional hyperparameters to tune, and convergence is not guaranteed.
3.1 GRADIENT ORIGIN NETWORKS
We propose a generative model that consists only of a decoding network, using empirical Bayes to approximate the posterior single step. That is, for some data point x and latent variable z Pz- we wish to find an approximation of p(z|x). Given some noisy observation Zo = z + A La) of then empirical Bayes can be applied to approximate z. Specifically, since we wish to approximate conditioned on x, we instead calculate 2x, the least squares estimate of p(z/x) (proof in Appendix A):
(20)
Zo +
V za log p(zo x).
(6)
Using Bayes' rule, log p(z)(x) can be written as log p(Zol>) = log p(x + log p(zo) log p(x).
Since log p(x) is normalising constant that does not affect the gradient, we can rewrite Equation in terms only of the decoding network and p(zo):
(zo) Zo + Vzo (log((x(zo) log p(zo)).
still remains, however, how to construct a noisy estimate of zo with no knowledge of z. If assume z follows known distribution, then it is possible to develop reasonable estimates. For instance, if we assume p(z) (zz 0, Ia) then we could sample from p(zu) = \Zzz; 2,) however this could be far from the true distribution of p(za 2) = NNan; z, Ia). Instead we propose initialising 2n at the origin since this is the distribution's 's mean. Initialising a constant position decreases the input variation and thus simplifies the optimisation procedure. Naturally, how p(x/z) is modelled affects 2x. While mian-field models result in 2x that are linear functions of x, conditional autoregressive models, for instance, result in non-linear 2x; multiple gradient steps also induce non-linearity, however, we show that a single step works well on high dimensional data suggesting that linear encoders, which normally do not scale to high dimensional data are effective in this case.
3.2 AUTOENCODING WITH GONS
Before exploring GONs as generative models, we discuss the case where the prior p(z) is unknown; such model is referred to as an autoencoder. As such, the distribution p(zo z) is also unknown thus it again unclear how we can construct noisy estimate of By training model end-to-end where Zo is chosen as the origin, however, prior is implicitly learned over such that it is reachable from zo- Although p(z) is unknown, we do not wish to impose prior on Zo: the term which enforces this is in Equation 7 is log P(Zo). so we can safely ignore this term and simply maximise the likelihood of the data given Zp. Our estimator of can therefore be defined simply as 2x(2n) = Zo + Vz, log p(x zo). which can otherwise be interpreted as single gradient descent step the conditional ligeliiel of the data. From this estimate, the data can be reconstructed by passing 2x through the decoder to parameterise p(x 2x) This procedure can be viewed more explicitly when using neural network F: &
IR' m output the mean of pxx(2x) parameterised by normal distribution; in this case the
3

---

Published as a conference paper at ICLR 2021
loss function is defined in terms of mean squared error loss L
Gx = CMSB
CMSB (x, F 2 F(zo)))).
(8)
The gradient computation thereby plays similar role to encoder, while F can be viewed as a decoder, with the outer loss term determining the overall reconstruction quality. Using a single network to perform both roles has the advantage of simplifying the overall architecture, avoiding the need to balance networks, and avoiding bottlenecks; this is demonstrated in Figure lb which provides visualisation of the GON process.
3.3
VARIATIONAL GONS
The variational approach can be naturally applied to GONs, allowing sampling in a single step while only requiring the generative network, reducing the parameters necessary. Similar to before, feedforward neural network F parameterises p(x while the expectation of p(z|x) is calculated with empirical Bayes. normal prior is assumed over p(z) thus Equation 7 can be written as:
2x Zo Zz log zo) + log.N 0, 2Ld)
(9)
where 2g as discussed in Section 3.1. While would be possible to use this estimate directly within constant-variance VAE (Ghosh et al., 2019), we opt incorporate the reparameterisatioo trick into the generative network as a stochastic layer, to represent the distribution over which x could be encoded to, using empirical Bayes estimate z. Similar to the autoencoding approach, could ignore log p(zu), however we find assuming a normally distributed Zy implicitly constrains z, aiding the optimisation procedure. Specifically, the forward pass of F is implemented follows: 2x (or 2n) is mapped by linear transformations to j((x) and or (2x) and the reparameterisation trick is applied, subsequently the further transformations formerly defined as F in the GON formulation are applied, providing parameters for for p(x (2x) Training is performed end-to-end, minimising the ELBO:
LVAB - -KL (u(2x) (( Ia)) + log p(x(2x).
(10)
These steps are shown in Figure 1d, which in practice has a simple implementation.
3.4 IMPLICIT GONS
In the field of implicit representation networks, the aim to learn a neural approximation of function & that satisfies an implicit equation of the form:
R(c, 4 Vs. v)...))= 00 c &(c).
(11)
where It's definition is problem dependent but often corresponds to a loss function. Equations with this structure arise in a myriad of fields, namely 3D modelling, image, video, and audio representation (Sitzmann et al., 2020b). In these cases, data samples x = {(c, Px(c))} can be represented in this form terms of coordinates E R² and the corresponding data at those points 4: E" - R". Due to the continuous nature of Φ data with large spatial dimensions can be represented much more efficiently than approaches using convolutions, for instance. Despite these benefits, representing distribution of data points using implicit methods is more challenging, leading to the use of hypernetworks which estimate the weights of an implicit network for each data point (Sitzmann et al., 2020a); this increases the number of parameters and adds significant complexity.
By applying the GON procedure to implicit representation networks, it possible learn a space of implicit functions without the need for additional networks. We assume there exist latent vectors & corresponding data samples; the concatentation of these latent variables with the data coordinates can therefore geometrically be seen as points on a manifold that describe the full dataset in keeping with the manifold hypothesis (Fefferman et al., 2016). An implicit Gradient Origin Network can be trained on this space to mimic & using a neural network F: Ra+ learning a space of implicit functions by modifying Equation 8:
R. thereby
Tx === f=/ L (c(),, F( 2 / [(*x(c), ((e F(c de,
(12)
where both integrations are performed over the space of coordinates and represents con- catenation (Figure Ic). Similar to the non- con-implicit approach, the computation latent vectors can be expressed as -V SIREN (Sitzmann et al.. 2020b), finding that it is capable of modelling both high and low frequency components in high dimensional spaces.
Vz, f [[( (C), F(c @ zo) dec. In particular, parameterise F with
4

---

Published as a conference paper at ICLR 2021
3.5 GON GENERALISATIONS
There are a number of interesting generalisations that make this approach applicable to other tasks. In Equations 8 and 12 we use the same C in both the inner term and outer term, however, as with variational GONs, it is possible to use different functions; through this, training a GON concurrently as a generative model and classifier is possible, or through some minor modifications to the loss involving the addition of the categorical cross entropy loss function _CCE to maximise the likelihood of classification, solely as a classifier:
Cx = ( [(x, F(zo))),
(13)
where y are the target labels and f is a single linear layer followed by softmax. Another possibility is modality conversion for translation tasks; in this case the inner reconstruction loss is performed on the source signal and the outer loss on the target signal.
3.6 JUSTIFICATION
Beyond empirical Bayes, we provide some additional analysis on why a single gradient step is in general sufficient information about data making it good encoder. Secondly, a good latent space should satisfy local consistency (Zhou al., 2003; Kamnitsas et al., 2018). GONs satisfy this since similar data points will have similar gradients due to the constant latent initialisation. As such, the network needs only to find an equilibrium where its prior the gradient operation, allowing for significant freedom. Finally, since GONs are trained by backpropagating through empirical Bayes, there are benefits to using an activation function whose second derivative non-zero.
encoder. Firstly, the gradient of the loss inherently offers substantial
4 RESULTS
We evaluate Gradient Origin Networks on a variety of image datasets: MNIST (LeCun et al., 1998), Fashion-MNiI (Xiao al., 2017). Small NORB (LeCun al., 2004), COIL-20 (Nane et al.. 1996). CIFAR-10 (Krizhevsky et al., 2009), CelebA (Liu et al., 2015), and LSUN Bedroom (Yu et al., 2015). Simple models are used: for small images, implicit GONs consist of approximately hidden layers of 256 units and convolutional GONs consist of 4 convolution layers with Batch Normalization (Ioffe Szegedy, 2015) and the ELU non-linearity (Clevert et al., 2016), for larger images the same general architecture is used, scaled up with additional layers; all training is performed with the Adam optimiser (Kingma & Ba, 2015). Error bars graphs represent standard deviations over three runs.
MNIST
Fashion-MNIST
Small NORB
COIL20
CIFAR-10
Single Step
GON (ours)
0.41
1.00
0.17
5.35+0.01
9.12
AE
1.33
1.702
0.73
6.05+10.11
12.24400055
Tied AE
0.06
2.452
0.93+0003
6.68+0.13
14.12+0.34
1 Step Detach
0.17
7.76+
1.84+0.01
15.72+0.70
30.17+
Multiple Steps
10 Step Detach
8.13=0.50
7.22
1.78+0.02
28.68+1.16 15.48
10 Step
0.02
1.01
0.17
5.36+0.011
9.19
GLO
0.70
1.78
0.61
3.27
9.12
Table 1: Validation reconstruction loss (summed squared error) over 500 epochs. For GLO, latents are assigned to data points and jointly optimised with the network. GONs significantly outperform other single step methods and achieve the lowest reconstruction error four of the five datasets.
MNIST
Fashion-MNIST
Small NORB
COTL.20
CIFAR-10
CelebA
VGON (ours)
1.06
3.30
2.34
3.44
5.85
5.41
VAE
1.15
3.23
2.54
3.63
5.94
5.59
Table 2:
Validation ELBO in bits/dim 1000 epochs (CelebA is trained over 150 epochs).
5

---

Published as a conference paper at ICLR 2021
4.1 QUANTITATIVE EVALUATION
A quantitative evaluation of the representation ability of GONs is performed in Table 1 against number baseline approaches. We compare against the single step methods: standard autoencoder, an autoencoder with tied weights (Gedeon, 1998), and GON with gradients detached from as well as the multi-step methods: 10 gradient descent steps per data point, with and without detaching gradients, and a GLO (Bojanowski data point and optimises them with gradient descent, therefore taking orders of magnitude longer to reconstruct validation data than other approaches. For the 10-step methods, learning rate of 0.1 applied as used in other literature (Nijkamp et al., 2019); the GLO is trained with MSE for consistency with the other approaches and do not project latents onto hypersphere proposed by the authors since in this experiment sampling is unimportant and this would handicap the approach. GONs achieve much lower validation loss than other single step methods and are competitive with the multi-step approaches; in fact, GONs achieve the lowest loss on four out of the five datasets.
al., 2018) which assigns persistent latent vector to each
Our variational GON is compared with VAE, whose decoder is the same as the GON, quantitatively in terms of ELBO on the test set in Table 2. We find that the representation ability of GONs is pertinent here also, allowing the variational GON to achieve lower ELBO on five out of the six datasets tested. Both models were trained with the original VAE formulation for faireess, however, we note that variations aimed at improving VAE sample quality such as 3 B-VAEs (Higgins et al., 2017) are also applicable to variational GONs to the same effect.
ablation study is performed, comparing convolutional GONs with autoencoders whose decoders have exactly the same architecture as the GONs (Figure 2a) and where the autoencoder decoders mirror their respective encoders. The mean reconstruction loss over the test set measured after each epoch for variety of latent sizes. Despite having half the number of parameters and linear encodings,
11022
10-2
10
1.5
AE Ixl
GON Ixl
GONT
Stop"
Train - GON Train
Steps'
Steps
AE Test
GON Test
Validation Loss
AE 4x4
GON 4x4
Valieatioon Loss
Steps'
Steps*
Recosstrnctinn
10
20
30
40
50
60
100
200
300
400
500
Training Egoch
Training Epoch
Training Epoch
(a) GONs achieve lower validation (b) Multiple latent update steps. *=grads detached, "noo detached.
(c) GONs overfit less than standard autoencoders.
loss than autoencoders.
Figure 2: Gradient Origin Networks trained on CIFAR-10 found to outperform autoencoders using exactly the same architecture without the encoder, requiring half the number of parameters.
LeakyRol.U
ELU
softplux
swish
-10-2
-10-2
-00
1.5
Recontraztion Loos
1.8
Reconsrrationn
Recomsraation
1.6
1.4
0.5
10
15
15
20
15
20
Training Epoch
Training Epoch
Training Epoch
(a) 512 latent variables
(b) 128 latent variables
(c) 32 latent variables
Figure 3: The impact activation function and number of latent variables on model performance for GON measured by comparing reconstruction losses through training.

---

Published as a conference paper at ICLR 2021
GONs achieve significantly lower reconstruction loss wide range of architectures. Further experiments evaluating the impact of latent size and model capacity are performed Appendix
Our hypothesis that for high dimensional datasets, single gradient step is sufficient when jointly optimised with the forwards pass is tested on the CIFAR-10 dataset Figure 2b. We observe negligible difference between a single step and multiple jointly optimised steps, in support of our hypothesis. Performing multiple steps greatly increases run-time so there seemingly no benefit in this case. Additionally, the importance of the joint optimisation procedure determined by detaching the gradients from z before reconstructing (Figure 2b); this results in markedly worse performance, even when in conjunction with multiple steps. In Figure 2c we assess whether the greater performance of GONs relative to autoencoders comes at the expense of generalisation; we find that the opposite true, that the discrepancy between reconstruction loss on the training and test sets is greater with autoencoders. This result extends to other datasets as can be seen in Appendix E.
Figure demonstrates the effect activation functions have on convolutional GON performance for different numbers of latent variables. Since optimising GONs requires computing second order derivatives, the choice of nonlinearity requires different characteristics to standard models. In particular, GONs prefer functions that are not only effective activation functions, but also whose second derivatives are non-zero, unlike ReLUs where ReLU"(z) = 0. The ELU non-linearity is effective with all tested architectures.
4.2 QUALITATIVE EVALUATION
The representation ability of implicit GONs is shown in Figure 4 where we train on large image datasets using a relatively small number of parameters. In particular, Figure 4a shows MNIST can be well-represented with just 4,385 parameters (a SIREN with 3 hidden layers each with 32 hidden units, and 32 latent dimensions). An advantage modelling data with implicit networks is that coordinates be at arbitrarily high resolutions. In Figure 5 we train on 32x32 images then reconstruct at 256x256. A significant amount of high frequency detail is modelled despite only seeing low resolution images. The structure of the implicit GON latent space shown by sampling latent codes from pairs of images, and then spherically interpolating between them to synthesise new samples (Figure 6). These samples shown capture variations shape (the shoes in Figure 6b), size, and rotation (Figure 6d).
We assess the quality of samples from variational GONs using convolutional models trained to
convergence in Figure 7. These are diverse and often contain fine details. Samples from an implicit GON trained with early stopping, as a simple alternative, can be found in Appendix D however this approach results in fine details being lost. GONs are also found to converge quickly; we plot
6
9
2
5
5
3
8
8
/
2
9
8
4
7
(a) 4,385 parameters (b) 297k parameters
(c) 270k parameters
(d) 270k parameters
(e) 396k parameters
Figure 4: Training implicit GONs with few parameters demonstrates their representation ability.
original
GON
original
GON
Figure 5: By training an implicit GON on 32x32 images, then sampling at 256x256, super-resolution possible despite never observing high resolution data.

---

Published as a conference paper at ICLR 2021
reconstructions multiple time points during the first minute of training (Figure 8). After only seconds of training on a single GPU, a large amount of signal information from MNIST is modelled.
In order to evaluate how well GONs represent high resolution natural images, convolutional GON on the LSUN Bedroom dataset scaled to 128x128 (Figure 9a). As with smaller, more simple data, find training always be extremely stable and consistent over wide range of hyperparameter settings. Reconstructions are of excellent quality given the simple network architecture. convolutional variational GON is also trained on the CelebA dataset scaled 64x64 (Figure 9b). Unconditional samples are somewhat blurry as commonly associated with traditional VAE models on complex natural images (Zhao al., 2017) but otherwise show wide variety.
train
5 DISCUSSION
Despite similarities with autoencoder approaches, the absence of encoding network offers several advantages. VAEs with overpowered decoders are known to ignore the latent variables (Chen al., 2017) whereas GONs only have network that equally serves both encoding and decoding functionality.. Designing inference networks for complicated decoders is not a trivial task (Vahdat & Kautz, 2020), however, inferring latent variables using GON simplifies this procedure. Similar to GONs, Normalizing Flow methods are also capable of encoding and decoding with a single
1
6
2
9
1
6
2 2 2 2 2
9
1
6
9
7
6
8
7
5
F
7 5
8
(a) MNIST
(b) Fashion-MNIST
(c) Small NORB
(d) COIL20
(c) CIFAR-10
Figure 6: Spherical linear interpolations between points in the latent space for trained implicit GONs
using different datasets (approximately 2-10 minutes training per dataset single GPU).
7
7
1
9
3
7
/ 9
4
3
7
0
6
4
2
7
2.
6
2
5
0
6
80
3
7
S
0
8
(a) 4,331 parameters
(b) 270k parameters
(c) 270k parameters
(d) 270k parameters
(e) 270k parameters
Figure 7: Random samples from a convolutional variational
GON with normally distributed latents.
70
7
0
3
4
7
0
3
4
7
0
3
4
7
0
3
4
7
0
3
4
3
5
6
3
5
6
3
5
6
3
5
6
3
5
1
6
7
2
8
5
7
2
8
5
7
2
8
5
7
2
8
5
7
2
8
5 9
9
6
9
9
9
6
9
9
9
6
9
9
9
6
9
9
9
6
9
(a) 300 steps
(b) 10s 1000 steps
(c) 30s 3000 steps
(d) 60s 6000 steps
(e) Ground Truth
Figure 8: Convergence of convolutional GONs with 74k parameters.
8

---

Published as a conference paper at ICLR 2021
(a) LSUN 128x128 Bedroom validation images (left) reconstructed by convolutional GON (right).
(b) Samples from convolutional variational GON trained on CelebA
Figure 9: GONs are able to represent high resolution complex datasets to a high degree of fidelity.
set of weights, however, they achieve this by restricting the network to be invertible. This requires considerable architectural restrictions that affect performance, make them less parameter efficient, and are unable to reduce the dimension of the data (Kingma Dhariwal 2018). Similarly, autoencoders with tied weights also encode and decode with single set of weights by using the transpose of the encoder's 's weight matrices in the decoder; this, however, is only applicable to simple architectures. GONs on the other hand use gradients as encodings which allow arbitrary functions to be used.
A number of previous works have used gradient- based computations to learn latent vectors however as far as aware, we are the first to use single gradient step jointly optimised with the feedforward pass, making it fundamentally different to these approaches. Latent encodings have been estimated for pre-trained generative models without encoders, namely Generative Adversarial Networks, using approaches such as standard gradient descent (Lipton & Tripathi, 2017: Zhu et al., 2016). A number of approaches have trained generative models directly with gradient- based inference (Han et al., 2017; Bojanowski et al., 2018; Zadeh et al., 2019); these assign latent vectors to data points and jointly learn them with the network parameters through standard gradient descent or Langevin dynamics. This is very slow, however, and convergence for unseen data samples is not guaranteed. Short run MCMC has also been applied (Nijkamp et al., 2020) however this still requires approximately 25 update steps. Since GONs train end-to-end, the optimiser can make use of the second order derivatives to allow for inference in a single step. Also of relevance model-agnostic meta-learning (Finn et al., 2017), which trains an architecture that few gradient descent steps are all that are necessary to new tasks. This is achieved by backpropagating through these gradients, similar to GONs.
In the case of implicit GONs, the integration terms in Equation 12 result in computation time that scales in proportion with the data dimension. This makes training slower for very high dimensional data, although we have not yet investigated Monte Carlo integration. In general GONs are stable and consistent, capable of generating quality samples with an exceptionally small number of parameters, and converge to diverse results with few iterations. Nevertheless, there are avenues to explore so as to improve the quality of samples and scale to larger datasets. In particular, it would be beneficial to focus on how to better sample these models, perform formal analysis on the gradients, and investigate whether the distance function could be improved to better capture fine details.
CONCLUSION
In conclusion, we have introduced a method based on empirical Bayes which computes the gradient of the data fitting loss with respect to the origin, and then jointly fits the data while learning this new point of reference in the latent space. The results show that this approach is able to represent datasets using small number of parameters with simple overall architecture, which has advantages in applications such as implicit representation networks. GONs are shown to converge faster with lower overall reconstruction loss than autoencoders, using the exact same architecture but without the encoder. Experiments show that the choice of non-linearity is important, as the network derivative jointly acts as the encoding function.

---

Published as a conference paper at ICLR 2021
REFERENCES
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the Latent Space of Generative Networks. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80, pp. 600-609, 2018.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and llya Sutskever. Generative Pretraining from Pixels. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational Lossy Autoencoder. In 5th International Conference Learning Representations, ICLR, 2017.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUUs). In 4th International Conference on Learning Representations, ICLR, 2016.
Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Tao Xia, and Lei Zhang. Second-order Attention Network for Single Image super-Resolution In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11065-11074, 2019.
Yilun Du and Igor Mordatch. Implicit Generation and Generalization in Energy-Basee -Based Models. In Neural Information Processing Systems (NeurIPS), 2019.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the Manifold Hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Leaming for Fast Adaptation of Deep Networks. Proceedings of the 34th International Conference on Machine Learning, 2017.
TD Gedeon. Stochastic Bidirectional Training. In SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 98CH36218), volume 2, pp. 1998.
Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From Variational to Deterministic Autoencoders. In 8th International Conference Learning Represen- futions, ICLR, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27. pp. 2672-2680. 2014.
Tian Han, Yang Lu, Song Chun Zhu, and Ying Nian Wu. Alternating Back- Propaprrpngttion for Generator Network. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp. 1976-1984, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In 5th International Conference on Learning Representations, ICLR, 2017.
Aapo Hyvärinen. Estimation of normalzze statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709, 2005.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, JCML, pp. 448 448-456, 2015.
Konstantinos Kamnitsas, Daniel C Castro, Loic Le Folgoe, Ian Walker, Ryutaro Tanno, Daniel Rueckert, Ben Glocker, Antonio Criminisi, and Aditya Nori. Semi-Supervised Learning via Compact Latent Space Clustering. Proceedings of the 35th International Conference on Machine Learning, 2018
10

---

Published as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In 3rd Interna- lional Conference on Learning Representations, ICLR, 2015.
Diederik P. Kingma and Max Welling, Auto-Encoding Variational Bayes. 2nd International Confer- ence on Learning Representations, ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible Ix1 Convolutions, In Advances in neural information processing systems, pp. 10215-10224, 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning Multiple Layers of Features from Tiny Images. Technical Report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11):2278-23)) -2224, 1998.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning Methods for Generie Object Recognition with Invariance to Pose and Lighting. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, CVPR, volume 2, pp. II-104, 2004.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural Speech Synthesis with Transformer Network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 6706-6713, 2019.
Zachary Lipton and Subarna Tripathi. Precise Recovery of Latent Vectors from Generative Adversarial Networks. In 5th International Conference on Learning Representations, ICLR, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Koichi Miyasawa. An Empirical Bayes Estimator of the Mean of Normal Population. Bulletin of the International Statistical Institute, 3814):181-188, 1961.
SA Nane, SK Nayar, and H Murase. Columbia Object Image Library: COIL-20. Technical Report CUCS-005-96, Columbia University, 1996.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning Non-Convergent Non- Persistent Short-Run MCMC Toward Energy-Based Model. In Neural Information Processing Systems (NeurIPS), pp. 5232-5242, 2019.
Erik Nijkamp, Bo Pang, Tian Han, Linqi Zhou, Song- CChun Zhu, and Ying Nian Wu. Learning Multi- layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference. stat, 1050: 17, 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-172 1754174, 1...274111 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. 32nd International Conference Machine Learning, ICML., 2:1530-1533 2015.
Herbert Robbins. An Empirical Bayes Approach Statistics. In Proc. Third Berkely Symp., volume 1, pp. 157-163, 1956.
Saeed Saremi and Aapo Hyvarinen. Neural Empirical Bayes. Journal of Machine Learning Research, 20:1-23, 2019.
Marwin HS Segler, Mike Preuss, and Mark Waller. Planning chemical syntheses with deep neural networks and symbolic AI. Nature, 555(7698):004-610, 2018.
11

---

Published as a conference paper at ICLR 2021
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. MetaSDF: Meta-learning Signed Distance Functions. In eural Information Processing Systems (NeurIPS), 2020a.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit Neural Representations with Periodic Activation Functions. In Neural Information Processing Systems (NeurIPS), 2020b.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Neural Information Processing Systems (NeurIPS), pp. 11918-11930, 2019.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Neural Information Processing Systems (NeurIPS), 2020.
Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks. In 33rd International Conference on Machine Learning, ICML 2016, volume 4, pp. 261 1-2620, 2016.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv preprint arXiv:1708.0774, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN:
Construction of Scage-Scale Image Dataset using Deep Learning with Humans in the Loop. arXiv preprint wrxivv/506.03366 2015.
Amir Zadeh, Yao-Chong Lim, Paul Pu Liang, and Louis-Philippe Morency. Variational Auto-Decoder. arXiv preprint arXiv: 1903.00800 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards Deeper Understanding of Variational Autoencoding Models. Proceedings of the 34th International Conference on Machine Learning, 2017.
Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. Learning with Local and Global Consistency. Advances in neural information processing systems, 16: 321-328, 2003.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexci A Efros. Generative Visual Manipula- tion on the Natural Image Manifold. In European Conference on Computer Vision, pp. 597-613, 2016.
A PROOF OF CONDITIONAL EMPIRICAL BAYES
For latent variables z ~ Pz, a noisy observation of 2, 2o = z + N(0, Ia), and data point x ~ Pd. we wish find an estimator of p(z|x). To achieve this, we condition the Bayes least squares estimator on x:
pl Z0, z/x) dz.
2x (2)) zp(z/zo, x)dz
(14)
pl (Zoolx)
Through the definition of the probabilistic chain rule and by marginalising out z. we can define p(zo) xx) [ p(zo/z, x) (z which can be simplified to p(zo 12) ())(2)x) since zo is dependent only on z. Writing this out fully, we obtain:
p(zolx)= = (2r)d/2|det({))) det(2) exp ( -
(-(zo (zo - Σ² - 2)/2 p(z x)d2. (15)
12

---

Published as a conference paper at ICLR 2021
Differentiating with respect to Za and multiplying both sides by E gives:
EV 2V2oP(Zol>3 = /(e zo)p x)p(z/x)
/ 20) 2 z/x]dz
(16)
f zp(zo, z/x)dz zop(zo|x).
(17)
After dividing both sides by p(zo(x) and combining with Equation 14 we get:
V VzoP(Zo x)
p(zo. 2|xx dz Zo =xx(zv) Zo-
Σ
=
(18)
p x))
p(zo(x)
Finally, this can be rearranged to give the single step estimator of
Vz,o(z))x)
(z() Zo E
Zn EV no log p(zn
(19)
x(Z
B COMPARISON WITH AUTOENCODERS
This section compares Gradient Origin Networks with autoencoders that exactly the same architecture, as described in the paper. All experiments are performed with the CIFAR-10 dataset. After each epoch the model is fixed and the mean reconstruction loss is measured over the training set.
In Figure 10a the depth of networks is altered by removing blocks thereby downscaling various resolutions. We find that GONs outperform autoencoders until latents have spatial size of 8x8 (where the equivalent GON now only has only 2 convolution layers). Considering the limit where neither model has any parameters, the latent space is the input data i.e. definition a GON (Equation 8) this becomes Vx C which simplifies to 0 = x which is contradiction. This is not concern in normal practice, evidenced by the results presented here.
Substituting the
Figure 10b explores the relationship between autoencoders and GONs when changing the number of convolution filters; GONs are found to outperform autoencoders in all cases. The discrepancy between loss curves decreases as the number filters increases likely due to the diminishing returns of providing more capacity to the GON when its loss is significantly closer to 0.
similar pattern is found when decreasing the latent space (Figure 10c); in this case the latent space likely becomes the limiting factor. With larger latent spaces GONs significantly outperform autoencoders, however, when the latent bottleneck becomes more pronounced this lead lessens.
-10-2
4R44x4
GON 4x4
AE f GON 18
ARA 512
5-2
Reconstrnction Loss
AB 6x6
GON 6x6
Roconstraction Loss
/E
AB ABk=122
GON 128
AE 1=188-1--000 GON
128
0.5
10
30
50
20
30
Training Epoch
Training Epoch
Training Epoch
(a) Encoding images to various spa- (b) Varying capacities by changing (c) Encoding images to variety tial dimensions.
the number of convolution filters f. latent space sizes.
Figure 10: Experiments comparing convolutional GONs with autoencoders CIFAR-10. where the GON uses exactly same architecture as the AF, without the encoder. (a) At the limit autoencoders tend towards the identity function whereas GONs unable to operate with parameters. As the number of network parameters increases (b) and the latent size decreases (c), the performance lead of GONs over AEs decreases due to diminishing return/bottleneckingg
13

---

Published as a conference paper at ICLR 2021
C INITIALISING Zn AT THE ORIGIN
We evaluate different initialisations of Zo Figure 11 by sampling Zo of( for a variety of standard deviations ø. The proposed approach (a 0) achieves the lowest reconstruction loss (Figure Ia); results for a > are similar, suggesting that the latent space is adjusted so Zo simulates the origin. An alternative parameterisation of z is to use use single gradient descent style step z 20 Z (Figure I lbb), however, losses are higher than the proposed GON initialisation.
(GON) -8
001
-4-
0.1
10
1.5-
Training Loss
1
Training
0.5
10
20
40
30
80
20
40
60
Training Epoch
Training Epoch
(a) z
V
V C
Figure 11: Training GONs with Zo sampled from variety of normal distributions with different standard deviations o, Zo o21) Approach (a) directly uses the negative gradients as encodings while approach (b) performs one gradient descent style step initialised at Zo.
D LATENT SPACE AND EARLY STOPPING
Figure 12 shows GON trained as a classifier where the latent space squeezed into 2D for visualisation (Equation 13) and Figure 13 shows that the negative gradients of GON after 800 steps of training are approximately normally distributed; Figure 14 shows that the latents typical convolutional VAE after 800 steps are significantly less normally distributed than GONs.
To obtain new samples with implicit GONs, we can use early stopping as a simple alternative to variational approaches. These samples (Figure 15) are diverse and capture the object shapes.
V
Figure 12:
2D latent space sam- Figure 13: Histogram la-
Figure 14: Histogram tra- ditional VAE latents after 800 steps.
ples of implicit GON clas- tent gradients after 800 implicit sifier trained on MNIST (class GON steps with a SIREN colours).
14

---

Published as a conference paper at ICLR 2021
2
6
9
8
8
7
J
6
6
1
9
4
9
0
5
6
3
0
8
9
7
8
9
9
0
6
8
3
7
0
(a) 4.385 parameters
(b) 70k parameters
(c) 270k parameters
(d) 270k parameters
(e) 270k parameters
Figure 15: GONs trained with early stopping can be sampled by approximating their latent space with multivariate normal distribution. These images show samples from an implicit GON trained with early stopping.
E GON GENERALISATION
In Figure 16 the training and test losses for GONs and autoencoders are plotted for variety of datasets. In all cases, GONs generalise better than their equivalent autoencoders while achieving lower losses with fewer parameters.
-10
-10-3
GON Train
GON Train
GON Triin GON Test AE Train
Rocestruction Loss
Reconntrccton Loes
AE Train
AE Train
Test
Reconsrucion
Test
500
100
200
300
400
500
Training Epoch
Training Epoch
Triining Epoch
(a) MNIST
(b) Fashion-MNIST MNIST
(c) Small NORB
Figure 16: The discrepancy between training and test reconstruction losses when using GON is smaller than equivalent autoencoders over a variety of datasets.
F AVAILABILITY
Source code for the convolutional GON, variational GON, and implicit GON is available under the MIT license GitHub at: httpss://gtttiii h/t/tttttbb/.t///////////ttttttt com/ ccom/wwx... ctt////w//////////////cc/ GON. This implementation uses PyTorch and all reported experiments use Nvidia RTX 2080 Ti GPU.
15

---

Published as a conference paper at ICLR 2021
G
SUPER-RESOLUTION
Additional super-sampled images obtained by training an implicit GON on 28x28 MNIST data then reconstructing test data a resolution of 256x256 are shown in Figure 17.
8
8
9
0
8
8
4
4
8
8
9
0
8
8
4
8
4
6
/
3
3
7
5
8
4
6
/
3
3
75
/
/
4
2045A360204$40
6
3
63306667
3
6
7
7
6
2
5
8 8
4
4
6
9
2
5
8
8
4
4
6
1
1
8
4
337
7
4
8
4
8
37
(a) Original data 28x28
(h) Implicit GON at 28x28
8 8
9
0
8
8
4
4
04663377
8
4
11673/7/
40218766
6
20452360
33016776
45884429
4 7
Figure 17: Super-sampling 28x28 MNIST test data 256x256 coordinates using implicit GON.
16


Reviews:
[{"review_id": "WVQgsB-Mi7q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a \"new\" inference method for autoencoder-type models, where the encoder is taken as a gradient of the decoder with respect to a zero-initialized latent variable. The method is evaluated for both a deterministic autoencoder and a VAE on toy image data (cifar10 being the most complex of them) and applied to convolutional decoder and to SIREN-type implicit representation networks. This is, for all intents and purposes, a single step iterative inference setup. In its VAE variant it is extremely similar to old-school iterative inference, albeit with a single gradient step. \n\nThe paper is very-well written and interesting. The method seems to be getting very good results,. Still, the paper seems to be rushed. The results are only on small scale and toyish datasets, and there are very few baselines. \n\nIn its current state I recommend rejection due to rather limited novelty (although it's cool to see that this type of inference works for implicit scene representations) and very limited evaluation. There are also very many links to existing literature that are not properly described. Let me elaborate.\n\nBaselines:\nTo determine the efficacy of this method, the authors would have to compare against some similar methods including:\n* old-school multi-step variational inference\n* semi-amortized variational inference\n* the proposed method with multiple gradient steps\n* the proposed method with detached gradient (as in: not use 2nd order gradients)\n* a fully-convolutional autoencoder with parameters tied between the encoder and decoder. This is for two reasons: a) this would reduce the number of parameters by half, making it more similar to GON, but also b) the transposed-convolution used in such a setup corresponds almost exactly to the gradient of the encoder, which is an idea very similar to GONs.\n\nMissing links to the literature:\n* the above fully-conv AE setup.\n* model-agnostic meta-learning (and related, e.g. CAVIA, LEO etc), where the \"latents\" are produced by single- or multi-step optimization.\n\nMissing experiments:\nWe would need more evidence to determine if such a simple method is useful. A good experiment would be e.g. on imagenet.\n\nFurther suggestions:\nSubfigures in fig2 and 3 (and most of figs in the appendix) use different scales on the Y axis. It would be easier to read the figures if the scaled were normalized within a single figure.\n\nUpdate: I've updated the score given the authors' response, see my comment below.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OTL9r-5XmOs", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new type of generative models with a new inference method of latent variables. Specifically, the gradient of latent variables with respect to zero vector is taken as the inferred latent variables. Based on this, the authors generalize the propose model to implicit and variational versions and demonstrate the models on image datasets.\n\nPros: the proposed method is easy and straightforward to implement. \n\nCons:\n1. The model assumption that the one step gradient from zero vector equals to latent vector is quite limited and greatly constrains the model expressiveness. A justification that such assumption is reasonable is badly needed.\n\n2. Formulation needs to be carefully checked. For example, Eqn 2 is not entirely correct to me. The second term should not be binary cross entropy as there is no categorical variable involved. Also, please avoid using abbreviations (L^BCE, L^CCE) at the first time to introduce them, which are confusing. \n\n3. Experimental results are not sufficient to demonstrate the efficacy. Need more quantitative analysis and experiments on more challenging datasets. \n\n4. The claim that it saves parameters compared to VAE is confusing. In the variational version, parametrizations of mu(x) and sigma(x) are also required. A principled way to very this claim is to show that with the variational version, the method could use much less parameters compared VAE while has the better synthesis quality. \n\nOverall, the method proposed in this paper is new and promising. However, given the current unclear formulation and lack of strong experimental results, I recommend a rejection. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EYWJekQyYEm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes GONs which seek to build a generative model with an “implicit” encoder that comes essentially for free with the use of a few re-parameterization tricks. The main idea being that existing generative models with an encoder are “redundant” in that the decoder itself has the ability to compute the gradient with respect to a latent vector, z, which itself can be thought of as the “encoding”. Since the choice of what initial latent vector to choose arises here, the paper advocates for simply choosing a z_0 which is a zero vector. In addition to the “explicit” formulation, there is also an implicit GON which is proposed that can generalize implicit generative models  (like SIREN) to entire distributions as opposed to a single data point, as they are currently used.\n\nOverall, I think this is very interesting work but incomplete. Considering GONs are a completely new category of generative models, it would greatly help to study each piece in more detail (theoretically or empirically) to establish what makes GONs successful, different, and how this improves our understanding of implicit representations in neural networks. \n\n\nStrengths:\n\n+ An interesting and novel formulation of encoding schemes from decoders that do not need any additional training or networks. \n+ The paper explores several different variants of GONs — from a variational alternative, implicit, and a classifier. Which greatly expands its scope of application in new problems. \n+ GONs generalize implicit generative models like SIRENs to work with an entire data distribution with very few parameters, which I think is a great benefit. This also naturally allows for variational alternatives, meaning we can sample from complex high dimensional distributions using very simple networks. \n+ The implicit GON also enables finer grid sampling in the input space, enabling its use in applications like super resolution naturally — but to any image from the training distribution. \n\n\nWeaknesses:\n\n* The paper is very dense in terms of ideas, and as such falls short in thoroughly evaluating all of them. For example, the paper contributes several ideas like GONs, implicit GONs, variational GONs, which is great but it would help if each one of those pieces were studied in some more detail so they can be compared and contextualized better with existing approaches. For example, in the formulation itself the GON loss is presented “as is”, but I think it warrants some more study. \n    * For example, why is just a single step “sufficient” to estimate “z”? Does the quality of “z” improve if you take multiple smaller steps? How stable is this for different datasets? The empirical studies show promise, that indeed this can work reasonably well in reconstructing different datasets, but it would greatly help to justify some of these choices further. \n    * In the explicit case, how important is the choice of “F” ? The choice of activation function is explored but what about the architecture/ number of parameters for a given dataset? \n* In all the experiments, the reconstruction losses are shown are for the training set, how do the validation set samples get reconstructed?  It’s not clear if GONs are so effective in reconstructing because they are memorizing the data? \n* How does the performance of GONs change as the size of the output space grows larger? For e.g. 128x128 or 256x256? \n* Some of the terminology is also confusing. What does it mean when you “overfit” to an entire distribution? I understand its usage for a single image, but it's not clear what this means for an entire dataset. Are the samples from Figure 4 all from the *same* trained GON? \n* Is Figure 7 from an explicit GON or an implicit GON? If its explicit, how are the number of parameters comparable to an implicitGON? Clearly an explicit model will have a lot more number of parameters. esp as the size of the images increase?\n* I really like and appreciate the variationalGON experiments. How do they compare with  standard VAEs? Can they recover CelebA 64x64 images? How would they compare on quantitative metrics like FID etc.?\n* In the super resolution experiment, can it super resolve *any* image from the distribution it was trained on? For e.g. in figure 5. is it just a matter of resampling the grid to 256x256 and running them through the pre-trained model for any sample from p(x)?\n\n---------- Update on the revised manuscript ----------\n\nI have read the new version of the paper and it reads a lot better. The new expanded methods section, and the definitions for different variations of GONs makes the paper much stronger and easier to understand. I appreciate and like the new experiments that show GONs capabilities on LSUN, comparisons with VAE on ELBO. \n\nMost of my concerns have been addressed in this version. I think this paper makes an interesting and novel contribution and I will raise my score accordingly. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new inference mechanism for latent variable models, by taking the derivative of log-likelihood with respect to a zero-valued vector. Initially, the reviewers raised concerns mostly regarding the limited experimentation and missing baselines. However, in the revised version, the authors addressed most of these concerns. 

Given that most reviewers are positive after the revision and since the proposed method is simple and interesting, I recommend accepting this paper.
Decision: Accept (Poster)