Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new “grandmother cell”-like memory mechanism for improving image generation performance in GANs. In short, this method clusters and stores activation vectors observed during training. Then at image generation time, activation vectors are augmented with the sum of the stored memories at the closest cluster. This seems to improve GAN performance for few-shot image generation tasks. The authors also visualize the learned memory clusters, which seems to reveal some semantic clustering.

main_review: This paper proposes a timely and interesting neural network memory mechanism, MoCA. The paper is fairly clear and well-written. However, the connections to neuroscience are weak (contra the introduction and discussion), and the image generation results raise some further questions that should probably be answered before recommending acceptance.

The authors’ central claim is that the proposed memory mechanism mechanism, which enables the network to memorize (transformed) activation vectors from images seen during training, similar to “grandmother cells” in human cortex, can enhance images generated by GANs. The connection to neuroscience here is tenuous at best — the authors draw inspiration from neuroscience, but I’m highly skeptical that the results are actually garnering any neuroscientific insight (to do that would probably require comparing these models to some form of brain recordings, but it is unclear how exactly that would be achieved with the current image-generation setup). So for the purpose of this review, I’m going to assume that this is simply a computer vision paper.

Quantitatively, the authors’ efforts seems successful: GANs trained using MoCA achieve better performance on few-shot learning than identical networks without MoCA. However, it is not clear that this comparison is entirely fair: do the StyleGAN2 and MoCA-StyleGAN2 (or FastGAN and MoCA-FastGAN) networks have the same number of parameters? This is not stated, but it seems like the MoCA layers will simply increase the parameter count, potentially inflating performance without revealing a real insight.

Relatedly, it seems important for the authors to test whether the MoCA layers are enabling the GANs to simply memorize the training images. In the example image figure (Figure 3), the authors should show the most similar image from the training set alongside each generated image.

The authors efforts to investigate the information learned in each memory cluster are important and interesting, but somewhat under-developed. In Figure 5 the authors show affinity maps for several memory clusters and images, and the results are somewhat convincing but also difficult to parse. I’m not sure how best to improve this analysis, but perhaps showing the same set of images for all the clusters would highlight the differences in cluster selectivity. 

Unfortunately the second memory cluster analysis (Figure 6) is much more difficult to understand. Here it seems like the authors are interpreting the tiny generated patches in the left-hand-side of each image in an attempt to understand what the semantic selectivity of each cluster is. While it seems clear that these patches come from different parts of the image (e.g. the left-most cluster is definitely selecting for sky in each image), the generated patches are very tiny and not very distinctive. I was left unsure about the claims made based on this Figure. I would recommend that the authors re-work this to be more clear about what they are showing, but I am afraid that I can’t really offer any specific and substantive recommendations.

summary_of_the_review: This paper proposes an interesting memory mechanism, and shows that it seems to help in some circumstances. However, questions about the quantitative evaluation, qualitative evaluation, and interpretation analyses leave me unsure about how much improvement the method really offers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a Memory Concept Attention mechanism to improve few-shot image generation quality. They showed that the prototype memory bank improves image synthesis quality, learns interpretable visual concept clusters and improves the robustness. 
main_review: Strengths:
The idea of using prototye memory concept is interesting, although it is not a new idea. For example, the prototypical network for few shot learning [1]. The authors showed that the proposed MoCA module can be used to enhance the geterator in several GAN based few-shot image generation models.

Weakness:
The authors should make the submission more clearer. The main concerns are listed below:
1) It is still an open question that how the "grandmother cell"  (if exists) codes information in the brain, e.g., population coding, sparse coding, or in other forms. The sparse activity of V1 neurons in the monkey brain doesn't mean they are grandmother cells. Activities of neurons in deeper layers (L3-5) may not be sparse. The authors should be more careful about this. 
2) the memory bank idea has been proposed in many other works (the implementation detailed may be different), the authors should compared the difference of their work and others. 
3) What do the prototype semantic cells and the prototype component cells correspond to in the neocortex? Is there a correspondence, or it is just for network design? Why not only use  $K_i$ to represent the prototype memory?
4) When transform A to a low-dimensional space, why do you use three functions $\theta(\cdot)$, $\phi(\cdot)$ and $\psi(\cdot)$? I only find $\phi(\cdot)$ in Fig.2. Also, in the main text, sometimes the authors use $\Phi$ and $\Psi$. All the terms should be defined clearly at the begining. 

[1] Snell, J., Swersky, K., & Zemel, R. S. (2017). Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.
summary_of_the_review: The authors carried out adequate experiments to show the proposed module (MoCA) is can be used to improve few-shot image generation quality and they linked the module to the grandmother cell in the brain. However, the memory bank idea is not new and the novelty is limited.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper proposes a novel prototype-based memory modulation layer (MoCA) to improve the generator network of a GAN. The target problem is few-shot image generation. Memory is arranged hierarchically into prototype semantic cells and prototype component cells. This design is loosely inspired by the recent discovery of "grandmother cells" in V1.

Experimental results show that in terms of FID score, using FastGAN base architecture, MoCA can bring 5.8% improvement on Animal Face Dog, 13.8% improvement on Obama, 21.7% improvement on ImageNet-100 and 12.4% improvement on COCO-300 dataset when using FastGAN as the baseline models. With StyleGAN2 base architecture, there was an 5.1% improvement on Animal Face Dog dataset, 8.1% improvement on Obama dataset, 14.1% improvement on ImageNet-100 dataset and 17.3% improvement on COCO-300 dataset.
main_review: Strengths
- Novelty. A novel prototype-based layer which significantly improves the respective augmented image generation models: FastGAN, StyleGAN2.
- Significance. The few-shot image generation is an important research topic.
- Relevance. Will be of interests to ICLR community, especially the vision researchers.
- Experimental Design. The performance improvements of adding MoCA layer to  two state-of-the-art architectures, FastGAN and StyleGAN2 were validated with 6 datasets of few-shot image synthesis: Animal-Face Dog, 100-Shot-Obama, ImageNet-100, COCO-300, CIFAR10 and Caltech-UCSD Birds (CUB). 
- Experimental Results. The consistent and somewhat significant improvements over the two baselines are impressive.
- Suitable ABLATION STUDIES were done.
- Analysis using visualization were performed to better understand the model.
- Writing is clear and well-structured.

Weaknesses
- Minor. Only 2 baselines were used to validate the novel design. The paper will be significantly strengthen with experiments on more baseline models.
- Minor. The modeling of the proposed method: MOCA has very limited correspondence in neuroscience. This is despite the paper's claim of being "inspired by neuroscience discoveries". As paper did not claim to be based on neuroscience, this is not a major concern.
summary_of_the_review: A well-written paper with novel and significant contributions to an important research topic. The design is clearly explained and the experimental results are excellent. While there are a couple of minor issues, the overall quality is sufficiently high for an acceptance by ICLR.
review_title: null

Meta-review: This paper uses prototype memories for learning generative models. Inspired by the finding that there is sparse activity and complex selectivity in the supragranular layers of every cortical region, even primary visual cortex, the authors propose to use prototype memories at each level of the hierarchy, which marks their work as novel. They show superior performance in few shot image generation tasks.

The reviewers' scores were borderline (5,5,8), making this a case that required some AC consideration. The reviewers generally agreed that the paper was relevant and interesting, though the two more negative reviewers had some concerns about (1) the tests used, (2) the interpretation relative to neuroscience data, and (3) the novelty. After reading through the paper, the reviews, and the rebuttal's, the AC felt that the authors had made a decent attempt at addressing items (1) and (2), and item (3) was ultimately a subjective question. The authors were reasonably clear about what marks their work as novel, and it is certainly not *exactly* the same as previous work. Altogether, given these considerations, the AC felt that this paper deserved to be accepted, given the reasonable attempts from the authors to respond to the reviewers' concerns and an average score above acceptance threshold (though the scores did not change post-rebuttal, it should be noted).
Decision: Accept (Poster)