Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper is a good idea. It is based on the notion of following the same line of reasoning as the prior work of MH-GAN. However, it makes the crucial advance of allowing the use of gradient information for the far more efficient HMC-type samplers.

The paper is able to do much of the sampling in the latent space, which enables the use of gradient-based sampling. One nitpick would be that I suspect there is a latent assumption that the mapping of z -> x is injective (but need not be bijective like in normalizing flows). However, this is not explicitly stated at all in the paper.
review_title: Good direction to move the area of Monte Carlo GAN refinement

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary
In this work, the authors propose coupling markov chain GAN (CMC-GAN), a technique that allows to draw GAN samples by running an MCMC chain that uses the generator to produce proposals, and the discriminator to compute acceptance probabilities.

Whereas prior work by Turner et al. (2019) uses an independent proposal, for each sample, the present work uses a small random perturbation of the last sample *in the latent space* to create the proposal. 
Goodfellow et al. 2014 showed that in the large data limit and large model capacity limit for absolutely continuous distributions, the output of the optimal discriminator represents encodes the density ratio of target and generated data.
Using this result, the authors of the present work derive a tractable expression for the acceptance probabilities that leads to an unbiased estimate of the target distribution in settings where the results of Goodfellow et al. 2014 hold.
They also show that an analogue acceptance criterion can be derived for the popular WGAN. 
The authors show experimental results on the synthetic swiss roll dataset as well as CIFAR10 and CELEBA with DCGAN and WGAN architecture, showing improved sample quality (measured by IS on the image datasets) compared to standard GAN and competing MCMC approaches, and improved acceptance rate compared to GAN. 

### Decision
I believe that proposes the most natural way for performing MCMC sampling on GANs. Both the dependent sampling based on the latent space and the acceptance mechanism based on the discriminator are natural and clean and, I'm sure will end up useful one way or another.
Where the paper is lacking, in my opinion, is in making a case *why* and *when* one should be using MCMC based methods in the first place when using GANs. 
While I do find the results on the swiss roll experiments are convincing, I am not convinced by the results on image datasets. 
The improvements seem to be minor, inconsistent, and on baselines performing worse than commonly reported in the literature.
Given how clean the proposed method itself is authors should improve the motivation by trying to find a more convincing application. I think it would be a wasted opportunity for what could be a very nice paper to accept it without such improvements.

### Questions/Suggestions

- The only motivation for the use of MCMC techniques is to not "waste the discriminator". Why should I expect the discriminator to be any better than the generator that it was trained with? Are there situations where one would naturally end up with a "better" discriminator than generator? Given that ordinary GAN sampling has 100 percent sample efficiency, I feel that a stronger argument in favor of using any MCMC based-technique in the first place is necessary.

- While I do find the experiments on swiss roll convincing, I am confused by the experiments on image GANs. **First**, the inception scores given in Table 2 seem significantly below those usually reported in the literature. **Second**, the experiments in Figure 3 show a minimal improvement of CMC-GAN compared to ordinary GAN sampling, in particular compared to the wide oscillation of IS over the different epochs. In particular, Figure 3 and Table 2 do not seem to be consistent. **Third**, it is my understanding that it does not make much sense to evaluate the IS on CELEBA. Usually, FID is used instead. 

- Recent work such as [Berard et al.](https://arxiv.org/abs/1906.04848), [Schafer et al.](https://arxiv.org/abs/1910.05852),  [Arjovsky and Bottou 2017](https://arxiv.org/abs/1701.04862) suggests that the behavior of discriminator training on image GANs might be radically different than in the idealized case analyzed by Goodfellow et al. in 2017. 
Could this be the reason why the results on image data sets fall short of those obtained in the absolutely continuous swiss roll dataset. Maybe techniques such as in [Dieng et al. 2019](https://arxiv.org/abs/1910.04302) could be useful?

- Although it has a somewhat different goal, it might still be useful to cite [Song et al.](https://arxiv.org/abs/1706.07561).

=============================================================================================

After discussion with the authors and reading the other reviews I still believe that the paper should not be accepted and therefore stay with my original review. While the authors have shown improvement over previous work (MH-GAN) using a very natural idea, this previous work in turn has not provided sufficient evidence that MCMC-GANs are useful.
I believe that we should not accept further MCMC-GAN papers, before this methodology has shown any improvement on a plausible use-case.
review_title: A great solution but I'm not yet fully convinced by the problem it is solving

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper proposes an MCMC sampling strategy for GANs. The idea is clear: for high-dimensional x, making a good proposal is difficult, so they propose to do that in the latent space.  Then they use a similar strategy as MH-GAN to compute a rejection strategy. The difference between the two methods is that proposal in MH-GAN does not depend on x while the proposed method does, and the argument is that it results in a higher acceptance rate. 

The idea is interesting, and the paper is relatively well-written, but the execution and experiment left a lot to be desired.

* One of the arguments of the paper (in the related work) the method overcomes sample low sample diversity. No metric is reported for that. The authors need to report LPIPS [1] and compare different methods. 

*  Fig 4 shows very poor diversity; all samples seem identical to each other, and honestly, it seems that the GAN collapsed. 

* The results in Fig 3 is also very strange. First of all, what is the point of Fig 3 (left)? All IS scores are roughly going up, but this figure is no comparison and does not show any if one method is doing better.  Fig 3 (right) is more concerning. One expects that as the epoch increases, the generator becomes better at generating samples. The acceptance rate improves (b/c G generates more realistic results and discriminator cannot tell them apart); why the acceptance rates are going down?

* Fig 2 is also presented in a confusing way. All methods except the second column from left almost perform the same and propose methods is in both columns (!!) What does each column, row mean? Given what I read in the related work, I expected to see the model's performance in the sample a multi-modal distribution. In fact, I want to see that: 100 Gaussian distributions centered at the 10x10 grids and small enough variance. I want to see how this method performs in recovering centers of the clusters (you can see examples in the literature). 

* Given that finally, the sampling strategy becomes very similar to the recent energy-based method. It is fair to say comparison is required. Yes, GAN is and the energy-based methods are different but the sampling strategy (the Langevian dynamic) are very similar and I would like to see how those two methods perform in term of quality of the samples. 


* Finally, if one views this method as a general strategy for generating samples using GAN, perhaps you can think of this approach as an approximate inference approach for inference in a complicated Baysian model. This paper does not focus on that and I don't expect authors to do any experiment for that but it can useful there.
 


[1] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The Unreasonable
Effectiveness of Deep Features as a Perceptual Metric. In CVPR, 2018.
review_title: good idea but not convincing experiments 

final_score: rating: 7: Good paper, accept
review_content: **Summary**
The paper proposes an MCMC based sampling mechanism for GANs. In contrast to earlier work, the proposal distribution is conditioning conditioned on the previous state (here in latent space), which is supposed to help sampling efficiency. This is achieved by a clever re-parametrization of intermediate steps of the MCMC chain. As an example, the authors provide a Langevin version (which uses gradient information) of their method.

I enjoyed reading this well-written paper and think the re-parameterized MCMC chain is a very neat idea that fits very naturally in the GAN framework. The paper is of quite incremental nature though: little to no theory contribution, only some incremental empirical benefits. I also have some doubts whether the comparisons are fair in terms of compute and effective sample size (more below).

Below are a few points to consider:

**Effective sample size**
What puzzles me here is the fact that while the independent proposal of Turner et al gets stuck in certain parts of the space, the Langevin updates of the proposed method (Figure 4) do have less variability. I assume the incremental improvement happen due to relatively small step sizes in the Langevin sampler? This would have a bad effect on the effective sample size. Does the MCMC chain eventually visits other parts of the space, or does it need to be restarted for every sample created? This does not seem to be true for the MH sampler, which despite its low acceptance rate produces samples from the entire space. Could the authors clarify this, and if possible provide measure of effective sample size for a fixed number of iterations?

**Compute**
The authors do neither discuss nor evaluate the computational load compared to other methods, especially performance metrics as a function of compute would be interesting. Or maybe the computational load is comparable for all methods?

**Title**
The term "coupled" is currently heavily used within the MCMC community, e.g. [2], and it denotes a different technique. I would suggest replacing it by "re-parameterized" which would be more accurate in that respect.

**"Proof" for convergence**
The abstract claims that convergence to the true data distribution is proved, but that is really stretching the term. All the paper contains is a valid MH ratio. But that alone does not prove convergence, for which the authors would need to go into the weeds of MCMC convergence, with conditions on the target distribution. This is especially true for the Langevin proposal, e.g. [1].

**Experiments**
I would suggest to move the Swiss roll dataset to the Appendix and replace it with something more interesting, preferably higher dimensional. This would make the paper stronger. In fact some analysis how all the different sampling methods compare as a function of dimensional (or "difficulty") would make it even more so.


[1] Roberts, G. O., & Tweedie, R. L. (1996). Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli.

[2] Jacob, P. E., O'Leary, J., & Atchadé, Y. F. (2017). Unbiased markov chain monte carlo with couplings. JRSS-B.
review_title: Neat paper!

Meta-review: The consensus among the reviewers is that this is a borderline paper: its main idea is sensible and natural. Unfortunately, while the reviewers appreciated the authors' responses to their comments, they felt that the paper failed to demonstrate the usefulness of the idea beyond toy-datasets. The latter would considerably strengthen this paper.
Decision: Reject