Gradient Origin Networks

Abstract: This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points. The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.

Reviews and author responses:

The paper proposes GONs which seek to build a generative model with an “implicit” encoder that comes essentially for free with the use of a few re-parameterization tricks. The main idea being that existing generative models with an encoder are “redundant” in that the decoder itself has the ability to compute the gradient with respect to a latent vector, z, which itself can be thought of as the “encoding”. Since the choice of what initial latent vector to choose arises here, the paper advocates for simply choosing a z_0 which is a zero vector. In addition to the “explicit” formulation, there is also an implicit GON which is proposed that can generalize implicit generative models  (like SIREN) to entire distributions as opposed to a single data point, as they are currently used.

Overall, I think this is very interesting work but incomplete. Considering GONs are a completely new category of generative models, it would greatly help to study each piece in more detail (theoretically or empirically) to establish what makes GONs successful, different, and how this improves our understanding of implicit representations in neural networks. 


Strengths:

+ An interesting and novel formulation of encoding schemes from decoders that do not need any additional training or networks. 
+ The paper explores several different variants of GONs — from a variational alternative, implicit, and a classifier. Which greatly expands its scope of application in new problems. 
+ GONs generalize implicit generative models like SIRENs to work with an entire data distribution with very few parameters, which I think is a great benefit. This also naturally allows for variational alternatives, meaning we can sample from complex high dimensional distributions using very simple networks. 
+ The implicit GON also enables finer grid sampling in the input space, enabling its use in applications like super resolution naturally — but to any image from the training distribution. 


Weaknesses:

* The paper is very dense in terms of ideas, and as such falls short in thoroughly evaluating all of them. For example, the paper contributes several ideas like GONs, implicit GONs, variational GONs, which is great but it would help if each one of those pieces were studied in some more detail so they can be compared and contextualized better with existing approaches. For example, in the formulation itself the GON loss is presented “as is”, but I think it warrants some more study. 
    * For example, why is just a single step “sufficient” to estimate “z”? Does the quality of “z” improve if you take multiple smaller steps? How stable is this for different datasets? The empirical studies show promise, that indeed this can work reasonably well in reconstructing different datasets, but it would greatly help to justify some of these choices further. 
    * In the explicit case, how important is the choice of “F” ? The choice of activation function is explored but what about the architecture/ number of parameters for a given dataset? 
* In all the experiments, the reconstruction losses are shown are for the training set, how do the validation set samples get reconstructed?  It’s not clear if GONs are so effective in reconstructing because they are memorizing the data? 
* How does the performance of GONs change as the size of the output space grows larger? For e.g. 128x128 or 256x256? 
* Some of the terminology is also confusing. What does it mean when you “overfit” to an entire distribution? I understand its usage for a single image, but it's not clear what this means for an entire dataset. Are the samples from Figure 4 all from the *same* trained GON? 
* Is Figure 7 from an explicit GON or an implicit GON? If its explicit, how are the number of parameters comparable to an implicitGON? Clearly an explicit model will have a lot more number of parameters. esp as the size of the images increase?
* I really like and appreciate the variationalGON experiments. How do they compare with  standard VAEs? Can they recover CelebA 64x64 images? How would they compare on quantitative metrics like FID etc.?
* In the super resolution experiment, can it super resolve *any* image from the distribution it was trained on? For e.g. in figure 5. is it just a matter of resampling the grid to 256x256 and running them through the pre-trained model for any sample from p(x)?

---------- Update on the revised manuscript ----------

I have read the new version of the paper and it reads a lot better. The new expanded methods section, and the definitions for different variations of GONs makes the paper much stronger and easier to understand. I appreciate and like the new experiments that show GONs capabilities on LSUN, comparisons with VAE on ELBO. 

Most of my concerns have been addressed in this version. I think this paper makes an interesting and novel contribution and I will raise my score accordingly. 



This paper proposes a new type of generative models with a new inference method of latent variables. Specifically, the gradient of latent variables with respect to zero vector is taken as the inferred latent variables. Based on this, the authors generalize the propose model to implicit and variational versions and demonstrate the models on image datasets.

Pros: the proposed method is easy and straightforward to implement. 

Cons:
1. The model assumption that the one step gradient from zero vector equals to latent vector is quite limited and greatly constrains the model expressiveness. A justification that such assumption is reasonable is badly needed.

2. Formulation needs to be carefully checked. For example, Eqn 2 is not entirely correct to me. The second term should not be binary cross entropy as there is no categorical variable involved. Also, please avoid using abbreviations (L^BCE, L^CCE) at the first time to introduce them, which are confusing. 

3. Experimental results are not sufficient to demonstrate the efficacy. Need more quantitative analysis and experiments on more challenging datasets. 

4. The claim that it saves parameters compared to VAE is confusing. In the variational version, parametrizations of mu(x) and sigma(x) are also required. A principled way to very this claim is to show that with the variational version, the method could use much less parameters compared VAE while has the better synthesis quality. 

Overall, the method proposed in this paper is new and promising. However, given the current unclear formulation and lack of strong experimental results, I recommend a rejection. 

We thank the reviewers for their valuable comments. During this rebuttal period we have made a very significant revision to the original paper, including the addition of Section 3.1, a new derivation from empirical Bayes (proof of conditional empirical bayes in Appendix A), added Table 1, showing quantitatively that GONs significantly outperform competing single-step methods, Table 2, demonstrating that variational GONs achieve substantially lower ELBO than VAEs (on 5/6 datasets) including large complex datasets (CelebA), Figure 9, showing qualitatively that GONs can well represent large complex datasets (added higher resolution LSUN Bedrooms and CelebA), Figure 2b, confirming that a single gradient step is sufficient, Figures 2c and 16, demonstrating GON generalisation, Figure 17, showing that GONs allow superresolution on test data, Table 1 and Figures 2a, 3, and 10 have been updated to include means/standard deviations (error envelopes) from multiple runs, alongside various new discussions.

Thank you for your kind words and for updating the score. We have implemented these suggested changes; error bars (as envelopes) representing standard deviation have been added to Table 1 and Figures 2a, 3, and 10 however we deemed it inappropriate for the other figures due to significant line overlapping impacting the presentation of these results.

This paper introduces a "new" inference method for autoencoder-type models, where the encoder is taken as a gradient of the decoder with respect to a zero-initialized latent variable. The method is evaluated for both a deterministic autoencoder and a VAE on toy image data (cifar10 being the most complex of them) and applied to convolutional decoder and to SIREN-type implicit representation networks. This is, for all intents and purposes, a single step iterative inference setup. In its VAE variant it is extremely similar to old-school iterative inference, albeit with a single gradient step. 

The paper is very-well written and interesting. The method seems to be getting very good results,. Still, the paper seems to be rushed. The results are only on small scale and toyish datasets, and there are very few baselines. 

In its current state I recommend rejection due to rather limited novelty (although it's cool to see that this type of inference works for implicit scene representations) and very limited evaluation. There are also very many links to existing literature that are not properly described. Let me elaborate.

Baselines:
To determine the efficacy of this method, the authors would have to compare against some similar methods including:
* old-school multi-step variational inference
* semi-amortized variational inference
* the proposed method with multiple gradient steps
* the proposed method with detached gradient (as in: not use 2nd order gradients)
* a fully-convolutional autoencoder with parameters tied between the encoder and decoder. This is for two reasons: a) this would reduce the number of parameters by half, making it more similar to GON, but also b) the transposed-convolution used in such a setup corresponds almost exactly to the gradient of the encoder, which is an idea very similar to GONs.

Missing links to the literature:
* the above fully-conv AE setup.
* model-agnostic meta-learning (and related, e.g. CAVIA, LEO etc), where the "latents" are produced by single- or multi-step optimization.

Missing experiments:
We would need more evidence to determine if such a simple method is useful. A good experiment would be e.g. on imagenet.

Further suggestions:
Subfigures in fig2 and 3 (and most of figs in the appendix) use different scales on the Y axis. It would be easier to read the figures if the scaled were normalized within a single figure.

Update: I've updated the score given the authors' response, see my comment below.

Thanks for addressing the majority of my comments, this is a much stronger paper now and I'm happy to increase my score. Some further suggestions:
- Table 1 should have the best results in boldface.
- It'd be great to have errors bars in Table 1 and 2 as well as most figures.
- In sec 3.6 "non-linearity function" should be "non-linear function".
- I like the newly-introduced argument (connections to MoE) of why GONs work well, though I must say that I am still baffled by how good it is. 



Thank you for your constructive feedback and excellent suggestions. We will address your comments point-by-point:

> “The results are only on small scale and toyish datasets”

In our latest update we have added experiments on larger scale more complex datasets: 128x128 LSUN Bedrooms for the GON, and CelebA 64x64 for the variational GON.

> “Baselines: To determine the efficacy of this method, the authors would have to compare against some similar methods including...”

For our non-variational approach, we have added quantitative comparisons with autoencoders+tied weights, our approach with detached gradients, our approach with multiple gradient descent steps, both with and without detached gradients, as suggested, as well as with a GLO (Bojanowski 2018) which assigns a latent vector to each data point and jointly optimises these with the network parameters (Table 1). We find that GONs achieve the lowest validation losses on 3 of the 5 datasets tested. Notably, all other single step approaches result in significantly high reconstruction loss.

We have also added quantitative comparisons with vanilla VAEs in Table 2, finding GONs to achieve lower ELBO on 5 of the 6 datasets. We aim to add comparisons with other variational approaches as suggested, in another update.

> “b) the transposed-convolution used in such a setup corresponds almost exactly to the gradient of the encoder, which is an idea very similar to GONs.”

While the gradient through a single convolution layer is indeed related to transposed convolutions, when convolutions are composed and/or interleaved with other functions, the gradient becomes much more complex. Indeed, the gradient of deep MLPs corresponds to a product of networks. Additionally, restricting the architecture to using tied-weights is not necessarily applicable to more complex architectures whereas using the gradient can be applied to any function.

> “Missing links to the literature…”

We have now integrated discussion of autoencoders with tied weights and the connections with model-agnostic meta-learning into the Discussion section.

> “Missing experiments: We would need more evidence to determine if such a simple method is useful. A good experiment would be e.g. on imagenet.”

We hope that the aforementioned additional quantitative experiments (Tables 1 and 2) including experiments on CelebA, and qualitative examples on LSUN Bedrooms assuage your concerns. Experiments have also been added to evaluate our claim that a single gradient step is sufficient and that GONs do not memorise datasets in Figures 2b and c respectively.

> “Further suggestions: Subfigures in fig2 and 3 (and most of figs in the appendix) use different scales on the Y axis. It would be easier to read the figures if the scaled were normalized within a single figure.”

All figures which we deemed legible when normalised with a single figure (previously Figures 2, 9, 10, and 11) have been changed as per your suggestion (now Figures 2 and 10).

> “Is Figure 7 from an explicit GON or an implicit GON? If its explicit, how are the number of parameters comparable to an implicitGON? Clearly an explicit model will have a lot more number of parameters. esp as the size of the images increase?”

Figure 7 is from an explicit GON, the caption has been updated to clarify this. The number of parameters used for implicit GONs and explicit GONs are shown in the captions of Figure 4 and 7 respectively. While not a direct comparison since Figure 7 is a variational GON and the numbers of parameters are chosen to be approximately comparable, this does provide some insight. We find that implicit models are able to better represent data when fewer parameters are available or when the image size is larger.

> “I really like and appreciate the variationalGON experiments. How do they compare with standard VAEs? Can they recover CelebA 64x64 images? How would they compare on quantitative metrics like FID etc.?”

In the latest update we have included quantitative comparisons between variational GONs and VAEs in terms of ELBO on the test set in Table 2. This includes evaluation on the CelebA 64x64 dataset. In 5 of the 6 datasets tested on, the GON approach achieves lower ELBO than the VAE. Samples from the variational GON trained on CelebA can also now be found in Figure 9b in order to assess this qualitatively.

> “In the super resolution experiment, can it super resolve any image from the distribution it was trained on? For e.g. in figure 5. is it just a matter of resampling the grid to 256x256 and running them through the pre-trained model for any sample from p(x)?”

Yes, it can super resolve any image thanks to the generalisation ability of GONs, and it is as simple as you state; Figure 17 has been updated to contain super-samples of images in the MNIST test set.

Thank you for your excellent comprehensive feedback. We have incorporated many of your suggestions in the most recent update. We will address your comments point-by-point:

> “The paper is very dense in terms of ideas, and as such falls short in thoroughly evaluating all of them. For example, the paper contributes several ideas like GONs, implicit GONs, variational GONs, which is great but it would help if each one of those pieces were studied in some more detail so they can be compared and contextualized better with existing approaches. For example, in the formulation itself the GON loss is presented “as is”, but I think it warrants some more study. ”

We have greatly expanded the method section, deriving our approach from empirical Bayes. Specifically, there is now a preliminaries section which introduces the concept of empirical Bayes and variational autoencoders in detail; additionally, the method section is now divided into sections, covering our contributions (GON, variational GON, implicit GON, and generalisations) in more detail as well as more thoroughly introducing the surrounding concepts.

> “For example, why is just a single step “sufficient” to estimate “z”? Does the quality of “z” improve if you take multiple smaller steps? How stable is this for different datasets? The empirical studies show promise, that indeed this can work reasonably well in reconstructing different datasets, but it would greatly help to justify some of these choices further. ”

Our new derivation from empirical Bayes shows that if we consider z_0 a noisy approximation of z, then we can use a single gradient step to calculate the expected value of p(z|x). As mentioned in our response to Reviewer 3, we also provide an explanation from a function approximating perspective, namely, the derivative of a deep MLP corresponds to a product of networks, allowing efficient modelling of high dimensional data. 

We have added experiments to evaluate the claim that a single step is sufficient in Figure 2b, finding that when jointly optimised, multiple steps offer no notable improvement over a single step, and training with gradients of z detached results in significantly worse performance. This can also be seen over a broad range of datasets, trained over long periods, in Table 1.

In terms of stability, we have observed no issues when training these models. Standard architectures are used and the Adam optimiser with default values. This is the case even on high resolutions data (up to 128x128 images tested). Additionally, we find them to be extremely consistent over multiple runs.

> “In the explicit case, how important is the choice of “F” ? The choice of activation function is explored but what about the architecture/ number of parameters for a given dataset?”

In Figure 10b the effect of number of parameters is explored. We find that GONs outperform their equivalent autoencoder in all cases. As the number of parameters is increased, this lead lessens due to diminishing returns. While we only use simple architectures, we have found all variations (e.g. upsampling, transposed convolutions, instance normalisation) to be effective.

> “In all the experiments, the reconstruction losses are shown are for the training set, how do the validation set samples get reconstructed? It’s not clear if GONs are so effective in reconstructing because they are memorizing the data?”

We have added extra experiments to assess this. In Figure 2c, Table 1, and Figure 16 training and validation losses are plotted for both GONs and their equivalent autoencoder; this demonstrates that GONs not only do not memorise the data, but appear to generalise better than autoencoders. 

> “How does the performance of GONs change as the size of the output space grows larger? For e.g. 128x128 or 256x256?”

We find the performance to be on par with smaller sized outputs. This is evaluated qualitatively by reconstructing 128x128 LSUN Bedrooms data with a convolutional GON in Figure 9a where a substantial amount of detail is modelled.

> “Some of the terminology is also confusing. What does it mean when you “overfit” to an entire distribution? I understand its usage for a single image, but it's not clear what this means for an entire dataset. Are the samples from Figure 4 all from the same trained GON?”

Thank you for pointing out this misnomer, the caption has been adjusted accordingly. This experiment is meant to compare with implicit representation networks which are trained on a single image. We show that GONs can represent whole datasets to a high degree of fidelity. To answer your question explicitly, the images in Figure 4 are indeed all from the same trained GON.

(Continued Below)

Thank you for your very helpful feedback, we will try to address your comments point-by-point:

> “The model assumption that the one step gradient from zero vector equals to latent vector is quite limited and greatly constrains the model expressiveness. A justification that such assumption is reasonable is badly needed.”

We have updated the paper, deriving our approach from empirical Bayes. In summary, for a latent variable z and data point x, if we have a noisy observation of z, i.e. z_0=z+N(0,I), then empirical Bayes’ allows us to obtain the expected value of p(z|x) using a single gradient step. When a normally distributed prior is assumed over z, then we can choose z_0 as the origin since it is the mean of p(z_0).

We also provide an explanation, from a function approximating perspective:
1. The gradient itself is a non-linear function that can approximate functions: the derivative of a deep MLP corresponds to a product of networks.
2. Using the gradient as an encoder offers good initialisation since it inherently provides an improved estimate of the latent vector.
3. Good latent spaces should satisfy local consistency (points close in latent space should be close in output space). Similar data points have similar gradients so this is satisfied. The exact gradient is thus relatively unimportant; the network’s prior must be the gradient operation but since the gradient is relatively unimportant, this does not severely restrict the network’s expressiveness.

> “Formulation needs to be carefully checked. For example, Eqn 2 is not entirely correct to me. The second term should not be binary cross entropy as there is no categorical variable involved. Also, please avoid using abbreviations (L^BCE, L^CCE) at the first time to introduce them, which are confusing.”

Thank you for pointing this out. The variational GON formulation has been altered to be more general.

> “Experimental results are not sufficient to demonstrate the efficacy. Need more quantitative analysis and experiments on more challenging datasets.”

We have added a number of additional experiments analysing the GON formulation including the effect of multiple gradient descent steps, confirming our hypothesis that a single step is sufficient (Figure 2b) and the ability for GONs to generalise (Figure 2c); qualitative experiments on more challenging datasets: reconstructions on LSUN Bedrooms (Figure 9a) and samples from a variational GON trained on CelebA (Figure 9b); and quantitative analysis comparing GONs with other approaches as suggested by Reviewer 1 where we find GONs to be competitive with multi-step approaches (Table 1) as well as a comparison between variational GONs and VAEs in terms of validation ELBO (Table 2). In a future update, more experiments will be added.

> “The claim that it saves parameters compared to VAE is confusing. In the variational version, parametrizations of mu(x) and sigma(x) are also required. A principled way to very this claim is to show that with the variational version, the method could use much less parameters compared VAE while has the better synthesis quality.”

To implement a variational GON we integrate the reparameterization trick into the decoder network. Specifically, the forward pass takes input z, is mapped by two linear layers to mu(z) and sigma(z), the reparameterization trick is applied, then the rest of the function is performed to obtain p(x|z). This allows us to use the GON update step to obtain z from the original z_0, while still parameterising mu and sigma. The parameters are thus reused in the derivative as the encoder so that there are just under half as many. We have attempted to clarify this in the variational GON section of the method. As suggested, we quantitatively verify this in Table 2 and find that the variational GON achieves lower validation ELBO than an equivalent VAE with almost twice as many parameters on 5 of the 6 datasets.

Meta-review: This paper presents a new inference mechanism for latent variable models, by taking the derivative of log-likelihood with respect to a zero-valued vector. Initially, the reviewers raised concerns mostly regarding the limited experimentation and missing baselines. However, in the revised version, the authors addressed most of these concerns. 

Given that most reviewers are positive after the revision and since the proposed method is simple and interesting, I recommend accepting this paper.

Decision: poster-presentations