Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Thank you for the feedback, and I have read it.

The authors claimed that they used techniques in [6] in which I am not an expert for this. However I cannot find the comparison that the authors mentioned in the feedback, so I am not sure if the claim is true.

I still recommend rejection for the paper, and as I said in the first review, the paper is not mature enough.

==== original review ===

The paper describes a generative model that replaces the GAN loss in the adversarial auto-encoder with MMD loss. Although the author claim the novelty as adding noise to the discriminator, it seems to me that at least for the RBF case it just does the following:
1. write down MMD as an integral probability metric (IPM)
2. say the test function, which originally should be in an RKHS, will be approximated using random feature approximations.

Although the authors explained the intuition a bit and showed some empirical results, I still don't see why this method should work better than directly minimising MMD. Also it is not preferred to look at the generated images and claim diversity, instead it's better to have some kind of quantitative metric such as the inception score.

Finally, given the fact that we have too many GAN related papers now, I don't think the innovation contained in the paper (which is using random features) is good enough to be published at ICLR. Also the paper is not clearly written, and I would suggest better not to copy-past paragraphs in the abstract and intro.

That said, I would welcome for the authors feedback and see if I have misunderstood something.
review_title: The paper is not mature enough to be accepted

final_score: rating: 3: Clear rejection
review_content: 
In this paper, the authors propose doubly stochastic adversarial autoencoder, which is essentially applying the doubly stochastic gradient for the variational form of maximum mean discrepancy. 

The most severe issue is lacking novelty. It is a straightforward combination of existing work, therefore, the contribution of this work is rare. 

Moreover, some of the claims in the paper are not appropriate. For example, using random features to approximate the kernel function does not bring extra stochasticity. The random features are fixed once sampled from the base measure of the corresponding kernel. Basically, you can view the random feature approximation as a linear combination of fixed nonlinear basis which are sampled from some distribution. 

Finally, the experiments are promising. However, to be more convincing, more benchmarks, e.g., cifar10/100 and CelebA, are needed. 
review_title: a straightforward extension of existing algorithms

final_score: rating: 2: Strong rejection
review_content: This manuscript explores the idea of adding noise to the adversary's play in GAN dynamics over an RKHS. This is equivalent to adding noise to the gradient update, using the duality of reproducing kernels. Unfortunately, the evaluation here is wholly unsatisfactory to justify the manuscript's claims. No concrete practical algorithm specification is given (only a couple of ideas to inject noise listed), only a qualitative one on a 2-dimensional latent space in MNIST, and an inconclusive one using the much-doubted Parzen window KDE method. The idea as stated in the abstract and introduction may well be worth pursuing, but not on the evidence provided by the rest of the manuscript.
review_title: Clear reject

Meta-review: The reviewers all outlined concerns regarding novelty and the maturity of this work. It would be helpful to clarify the relation to doubly stochastic kernel machines as opposed to random kitchen sinks, and to provide more insight into how this stochasticity helps. Finally, the approach should be tried on more difficult image datasets.
Decision: Reject