Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a new type of generative models with a new inference method of latent variables. Specifically, the gradient of latent variables with respect to zero vector is taken as the inferred latent variables. Based on this, the authors generalize the propose model to implicit and variational versions and demonstrate the models on image datasets.

Pros: the proposed method is easy and straightforward to implement. 

Cons:
1. The model assumption that the one step gradient from zero vector equals to latent vector is quite limited and greatly constrains the model expressiveness. A justification that such assumption is reasonable is badly needed.

2. Formulation needs to be carefully checked. For example, Eqn 2 is not entirely correct to me. The second term should not be binary cross entropy as there is no categorical variable involved. Also, please avoid using abbreviations (L^BCE, L^CCE) at the first time to introduce them, which are confusing. 

3. Experimental results are not sufficient to demonstrate the efficacy. Need more quantitative analysis and experiments on more challenging datasets. 

4. The claim that it saves parameters compared to VAE is confusing. In the variational version, parametrizations of mu(x) and sigma(x) are also required. A principled way to very this claim is to show that with the variational version, the method could use much less parameters compared VAE while has the better synthesis quality. 

Overall, the method proposed in this paper is new and promising. However, given the current unclear formulation and lack of strong experimental results, I recommend a rejection. 
review_title: Lack of solid formulation and strong experiments

final_score: rating: 7: Good paper, accept
review_content: The paper proposes GONs which seek to build a generative model with an “implicit” encoder that comes essentially for free with the use of a few re-parameterization tricks. The main idea being that existing generative models with an encoder are “redundant” in that the decoder itself has the ability to compute the gradient with respect to a latent vector, z, which itself can be thought of as the “encoding”. Since the choice of what initial latent vector to choose arises here, the paper advocates for simply choosing a z_0 which is a zero vector. In addition to the “explicit” formulation, there is also an implicit GON which is proposed that can generalize implicit generative models  (like SIREN) to entire distributions as opposed to a single data point, as they are currently used.

Overall, I think this is very interesting work but incomplete. Considering GONs are a completely new category of generative models, it would greatly help to study each piece in more detail (theoretically or empirically) to establish what makes GONs successful, different, and how this improves our understanding of implicit representations in neural networks. 


Strengths:

+ An interesting and novel formulation of encoding schemes from decoders that do not need any additional training or networks. 
+ The paper explores several different variants of GONs — from a variational alternative, implicit, and a classifier. Which greatly expands its scope of application in new problems. 
+ GONs generalize implicit generative models like SIRENs to work with an entire data distribution with very few parameters, which I think is a great benefit. This also naturally allows for variational alternatives, meaning we can sample from complex high dimensional distributions using very simple networks. 
+ The implicit GON also enables finer grid sampling in the input space, enabling its use in applications like super resolution naturally — but to any image from the training distribution. 


Weaknesses:

* The paper is very dense in terms of ideas, and as such falls short in thoroughly evaluating all of them. For example, the paper contributes several ideas like GONs, implicit GONs, variational GONs, which is great but it would help if each one of those pieces were studied in some more detail so they can be compared and contextualized better with existing approaches. For example, in the formulation itself the GON loss is presented “as is”, but I think it warrants some more study. 
    * For example, why is just a single step “sufficient” to estimate “z”? Does the quality of “z” improve if you take multiple smaller steps? How stable is this for different datasets? The empirical studies show promise, that indeed this can work reasonably well in reconstructing different datasets, but it would greatly help to justify some of these choices further. 
    * In the explicit case, how important is the choice of “F” ? The choice of activation function is explored but what about the architecture/ number of parameters for a given dataset? 
* In all the experiments, the reconstruction losses are shown are for the training set, how do the validation set samples get reconstructed?  It’s not clear if GONs are so effective in reconstructing because they are memorizing the data? 
* How does the performance of GONs change as the size of the output space grows larger? For e.g. 128x128 or 256x256? 
* Some of the terminology is also confusing. What does it mean when you “overfit” to an entire distribution? I understand its usage for a single image, but it's not clear what this means for an entire dataset. Are the samples from Figure 4 all from the *same* trained GON? 
* Is Figure 7 from an explicit GON or an implicit GON? If its explicit, how are the number of parameters comparable to an implicitGON? Clearly an explicit model will have a lot more number of parameters. esp as the size of the images increase?
* I really like and appreciate the variationalGON experiments. How do they compare with  standard VAEs? Can they recover CelebA 64x64 images? How would they compare on quantitative metrics like FID etc.?
* In the super resolution experiment, can it super resolve *any* image from the distribution it was trained on? For e.g. in figure 5. is it just a matter of resampling the grid to 256x256 and running them through the pre-trained model for any sample from p(x)?

---------- Update on the revised manuscript ----------

I have read the new version of the paper and it reads a lot better. The new expanded methods section, and the definitions for different variations of GONs makes the paper much stronger and easier to understand. I appreciate and like the new experiments that show GONs capabilities on LSUN, comparisons with VAE on ELBO. 

Most of my concerns have been addressed in this version. I think this paper makes an interesting and novel contribution and I will raise my score accordingly. 


review_title: An interesting new perspective on generative modeling and implicit representation learning, but incomplete in its execution.

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a "new" inference method for autoencoder-type models, where the encoder is taken as a gradient of the decoder with respect to a zero-initialized latent variable. The method is evaluated for both a deterministic autoencoder and a VAE on toy image data (cifar10 being the most complex of them) and applied to convolutional decoder and to SIREN-type implicit representation networks. This is, for all intents and purposes, a single step iterative inference setup. In its VAE variant it is extremely similar to old-school iterative inference, albeit with a single gradient step. 

The paper is very-well written and interesting. The method seems to be getting very good results,. Still, the paper seems to be rushed. The results are only on small scale and toyish datasets, and there are very few baselines. 

In its current state I recommend rejection due to rather limited novelty (although it's cool to see that this type of inference works for implicit scene representations) and very limited evaluation. There are also very many links to existing literature that are not properly described. Let me elaborate.

Baselines:
To determine the efficacy of this method, the authors would have to compare against some similar methods including:
* old-school multi-step variational inference
* semi-amortized variational inference
* the proposed method with multiple gradient steps
* the proposed method with detached gradient (as in: not use 2nd order gradients)
* a fully-convolutional autoencoder with parameters tied between the encoder and decoder. This is for two reasons: a) this would reduce the number of parameters by half, making it more similar to GON, but also b) the transposed-convolution used in such a setup corresponds almost exactly to the gradient of the encoder, which is an idea very similar to GONs.

Missing links to the literature:
* the above fully-conv AE setup.
* model-agnostic meta-learning (and related, e.g. CAVIA, LEO etc), where the "latents" are produced by single- or multi-step optimization.

Missing experiments:
We would need more evidence to determine if such a simple method is useful. A good experiment would be e.g. on imagenet.

Further suggestions:
Subfigures in fig2 and 3 (and most of figs in the appendix) use different scales on the Y axis. It would be easier to read the figures if the scaled were normalized within a single figure.

Update: I've updated the score given the authors' response, see my comment below.
review_title: Very interesting paper and findings, but seems somewhat rushed.

Meta-review: This paper presents a new inference mechanism for latent variable models, by taking the derivative of log-likelihood with respect to a zero-valued vector. Initially, the reviewers raised concerns mostly regarding the limited experimentation and missing baselines. However, in the revised version, the authors addressed most of these concerns. 

Given that most reviewers are positive after the revision and since the proposed method is simple and interesting, I recommend accepting this paper.
Decision: Accept (Poster)