Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: [Overview]

In this paper, the authors proposed a novel model called MemoryGAN, which integrates memory network with GAN. As claimed by the authors, MemoryGAN is aimed at addressing two problems of GAN training: 1) difficult to model the structural discontinuity between disparate classes in the latent space; 2) catastrophic forgetting problem during the training of discriminator about the past synthesized samples by the generator. It exploits the life-long memory network and adapts it to GAN. It consists of two parts, discriminative memory network (DMN) and Memory Conditional Generative Network (MCGN). DMN is used for discriminating input samples by integrating the memory learnt in the memory network, and MCGN is used for generating images based on random vector and the sampled memory from the memory network. In the experiments, the authors evaluated memoryGAN on three datasets, CIFAR-10, affine-MNIST and Fashion-MNIST, and demonstrated the superiority to previous models. Through ablation study, the authors further showed the effects of separate components in memoryGAN. 

[Strengths]

1. This paper is well-written. All modules in the proposed model and the experiments were explained clearly. I enjoyed much to read the paper.

2. The paper presents a novel method called MemoryGAN for GAN training. To address the two infamous problems mentioned in the paper, the authors proposed to integrate a memory network into GAN. Through memory network, MemoryGAN can explicitly learn the data distribution of real images and fake images. I think this is a very promising and meaningful extension to the original GAN. 

3. With MemoryGAN, the authors achieved best Inception Score on CIFAR-10. By ablation study, the authors demonstrated each part of the model helps to improve the final performance.

[Comments]

My comments are mainly about the experiment part:

1. In Table 2, the authors show the Inception Score of images generated by DCGAN at the last row. On CIFAR-10, it is ~5.35. As the authors mentioned, removing EM, MCGCN and Memory will result in a conventional DCGAN. However, as far as I know, DCGAN could achieve > 6.5 Inception Score in general.  I am wondering what makes such a big difference between the reported numbers in this paper and other papers?

2. In the experiments, the authors set N = 16,384, and M = 512, and z is with dimension 16. I did not understand why the memory size is such large. Take CIFAR-10 as the example, its training set contains 50k images. Using such a large memory size, each memory slot will merely count for several samples. Is a large memory size necessary to make MemoryGAN work? If not, the authors should also show ablated study on the effect of different memory size; If it is true, please explain why is that. Also, the authors should mention the training time compared with DCGAN. Updating memory with such a large size seems very time-consuming.

3. Still on the memory size in this model. I am curious about the results if the size is decreased to the same or comparable number of image categories in the training set. As the author claimed, if the memory network could learn to cluster training data into different category, we should be able to see some interesting results by sampling the keys and generate categoric images.

4. The paper should be compared with InfoGAN (Chen et al. 2016), and the authors should explain the differences between two models in the related work. Similar to MemoryGAN, InfoGAN also did not need any data annotations, but could learn the latent code flexibly.

[Summary]

This paper proposed a new model called MemoryGAN for image generation. It combined memory network with GAN, and achieved state-of-art performance on CIFAR-10. The arguments that MemoryGAN could solve the two infamous problem make sense. As I mentioned above, I did not understand why the authors used such large memory size. More explanations and experiments  should be conducted to justify this setting. Overall, I think MemoryGAN opened a new direction of GAN and worth to further explore.


review_title: Review from AnonReviewer3

final_score: rating: 6: Marginally above acceptance threshold
review_content: In summary, the paper introduces a memory module to the GANs to address two existing problems: (1) no discrete latent structures and (2) the forgetting problem. The memory provides extra information for both the generation and the discrimination, compared with vanilla GANs. Based on my knowledge, the idea is novel and the Inception Score results are excellent. However, there are several major comments should be addressed, detailed as follows:

1. The probabilistic interpretation seems not correct.

According to Eqn (1), the authors define the likelihood of a sample x given a slot index c as p(x|c=i) = N(q; K_i, sigma^2), where q is the normalized output of a network mu given x. It seems that this is not a well defined probability distribution because the Gaussian distribution is defined over the whole space while the support of q is restricted within a simplex due to the normalization. Then, the integral over x should be not equal to 1 and hence all of the probabilistic interpretation including the equations in the Section 3. and results in the Section 4.1. are not reliable. I'm not sure whether there is anything misunderstood because the writing of the Section 3 is not so clear. 

2. The writing of the Section 3 should be improved.

Currently, the Section 3 is not easy to follow for me due to the following reasons. First, there lacks a coherent description of the notations. For instance, what's the difference between x and x', used in Section 3.1.1 and 3.1.2 respectively? According to the paper, both denote a sample. Second, the setting is somewhat unclear. For example,  it is not natural to discuss the posterior without the clear definition of the likelihood in Eqn (1). Third, a lot of details and comparison with other methods should be moved to other parts and the summary of the each part should be stated explicitly and clearly before going into details.

3. Does the large memory hurt the generalization ability of the GANs?

First of all, I notice that the random noise is much lower dimensional than the memory, e.g. 2 v.s. 256 on affine-MNIST. Does such large memory hurt the generalization ability of GANs? I suspect that most of the information are stored in the memory and only small change of the training data is allowed. I found that the samples in Figure 1 and Figure 5 are very similar and the interpolation only shows a very small local subspace near by a training data, which cannot show the generalization ability. Also note that the high Inception Score cannot show the generalization ability as well because memorizing the training data will obtain the highest score. I know it's hard to evaluate a GAN model but I think the authors can at least show the nearest neighbors in the training dataset and the training data that maximizes the activation of the corresponding memory slot together with the generated samples to see the difference.

Besides, personally speaking, Figure 1 is not so fair because a MemoryGAN only shows a very small local subspace near by a training data while the vanilla GAN shows a large subspace, making the quality of the generation different. The MemoryGAN also has failure samples in the whole latent space as shown in Figure 4.

Overall, I think this paper is interesting but currently it does not reach the acceptance threshold.

I change the rating to 6 based on the revised version, in which most of the issues are addressed.
review_title: Comments on the probabilistic interpretation, writing and the generalization ability

final_score: rating: 6: Marginally above acceptance threshold
review_content: MemoryGAN is proposed to handle structural discontinuity (avoid unrealistic samples) for the generator, and the forgetting behavior of the discriminator. The idea to incorporate memory mechanism into GAN is interesting, and the authors make nice interpretation why this needed, and clearly demonstrate which component helps (including the connections to previous methods).   

My major concerns:

Figure 1 is questionable in demonstrating the advantage of proposed MemoryGAN. My understanding is that four z's used in DCGAN and MemoryGAN are "randomly sampled" and fixed, interpolation is done in latent space, and propagate to x to show the samples.  Take MNIST for example, It can be seen that the DCGAN has to (1) transit among digits in different classes, while MemoryGAN only (2) transit among digits in the same class. Task 1 is significantly harder than task 2, it is not surprise that DCGAN generate unrealistic images. A better experiment is to fix four digits from different class at first, find their corresponding latent codes, do interpolation, and propagate back to sample space to visualize results. If the proposed technique can truly handle structural discontinuity, it will "jump" over the sample manifold from one class to another, and thus avoid unrealistic samples. Also, the current illustration also indicates that the generated samples by MemoryGAN is not diverse.

It seems the memory mechanism can bring major computational overhead, is it possible to provide the comparison on running time?

To what degree the MemoryGAN can handle structural discontinuity? It can be seen from Table 2 that larger improvement is observed when tested on a more diverse dataset. For example, the improvement gap from MNIST to CIFAR is larger. If the MemoryGAN can truly deal with structural discontinuity, the results on generating a wide range of different images for ImageNet may endow the paper with higher impact.

The authors should consider to make their code reproducible and public. 


Minor comments:

In Section 4.3, Please fix "Results in 2" as "Results in Table 2".



review_title: An interesting idea with clear demonstration

Meta-review: 
I am going to recommend acceptance of this paper despite being worried about the issues raised by reviewer 1.  In particular,

1:  the best possible inception score would be obtained by copying the training dataset
2:  the highest visual quality samples would be obtained by copying the training dataset
3:  perturbations (in the hidden space of a convnet) of training data might not be perturbations in l2, and so one might not find a close nearest neighbor with an l2 search
4:  it has been demonstrated in other works that perturbations of convnet features of training data (e.g. trained as auto-encoders) can make convincing "new samples"; or more generally, paths between nearby samples in the hidden space of a convnet can be convincing new samples.

These together suggest the possibility that the method presented is not necessarily doing a great job as a generative model or as a density model (it may be, we just can't tell...), but it is doing a good job at hacking the metrics (inception score, visual quality).      This is not an issue with only this paper, and I do not want to punish the authors of this papers for the failings of the field; but this work, especially because of its explicit use of training examples in the memory,  nicely exposes the deficiencies in our community's methodology for evaluating GANs and other generative models.


Decision: Accept (Poster)