Prototype memory and attention mechanisms for few shot image generation
Authors: Tianqin Li
Zijie Li
Andrew Luo
Harold Rockwell
Amir Barati Farimani
Tai Sing Lee
Venue: ICLR 2022 Poster (2022)
Keywords: neuroscience
deep learning

Abstract: Recent discoveries indicate that the neural codes in the primary visual cortex (V1) of macaque monkeys are complex, diverse and sparse. This leads us to ponder the computational advantages and functional role of these “grandmother cells." Here, we propose that such cells can serve as prototype memory priors that bias and shape the distributed feature processing within the image generation process in the brain. These memory prototypes are learned by momentum online clustering and are utilized via a memory-based attention operation, which we define as Memory Concept Attention (MoCA). To test our proposal, we show in a few-shot image generation task, that having a prototype memory during attention can improve image synthesis quality, learn interpretable visual concept clusters, as well as improve the robustness of the model. Interestingly, we also find that our attentional memory mechanism can implicitly modify the horizontal connections by updating the transformation into the prototype embedding space for self-attention. Insofar as GANs can be seen as plausible models for reasoning about the top-down synthesis in the analysis-by-synthesis loop of the hierarchical visual cortex, our findings demonstrate a plausible computational role for these “prototype concept" neurons in visual processing in the brain.

Published as a conference paper at ICLR 2022
PROTOTYPE MEMORY AND ATTENTION MECHANISMS FOR FEW SHOT IMAGE GENERATION
Tianqin Li', Zijie Li', Andrew Luo, Harold Rockwell, Amir Barati Farimani, Tai Sing Lee Carnegie Mellon University (tianqinl, zijiel, afluo, hrockwel, afariman, taislee) @andrew. cmu. edu
ABSTRACT
Recent discoveries indicate that the neural codes in the superficial layers of the primary visual cortex (V1) of macaque monkeys are complex, diverse and super- sparse. This leads us to ponder the computational advantages and functional role of these "grandmother cells." " Here, we propose that such cells can serve as prototype memory priors that bias and shape the distributed feature processing during the image generation process the brain. These memory prototypes are learned by momentum online clustering and are utilized through memory- baaser attention operation. Integrating this mechanism, we propose Memory Concept Attention (MoCA) to improve few shot image generation quality. We show that having a prototype memory with attention mechanisms can improve image synthesis quality, learn interpretable visual concept clusters, and improve the robustness of the model. Our results demonstrate the feasibility of the idea that these super-sparse complex feature detectors can serve prototype memory priors for modulating the image synthesis processes in the visual system.
INTRODUCTION
Recent neurophysiological findings based on calcium imaging have revealed that many neurons in the superficial layers of V1 are strongly tuned to complex local patterns, rather than simple oriented edges or bars. These complex neurons exhibit far stronger responses (a 3-5 fold increase) when exposed to their preferred patterns than when exposed to simple grating and bar-like stimuli. The high degree of specificity in these neurons' selectivity suggests that they might serve as specific pattern detectors (Tang et al., 2018a), Due the selectivity towards complex stimuli, the population response of these V1 neurons is extremely sparse. Only 4-6 out of roughly 1000 neurons respond strongly to any given pattern or natural stimulus (Tang al., 2018b). This finding is reminiscent of an earlier study that found similarly sparse encoding of concepts in the hippocampus (Quiroga et al., 2005). Recent data in V4 also indicate the existence of cells with a similarly high degree of sparsity". Thus, we theorize such cells should exist in every level of the hierarchical visual system. We called these highly selective
feature detectors "grandmother neurons" " to highlight their possible explicit encoding of specific prototypes, even though reality, a prototype is likely represented by a sparse cluster of neurons rather than a single cell. Neurons in different layers of each visual area exhibit different degrees of response sparsity, complementing various functions. The observations of this diverse set of super-sparsely responding feature detectors raise the following question: what are the possible computational benefits and rationales of having such neurons in the early visual cortex?
another in
In this paper, we hypothesize that these "grandmother neurons" can serve as prototype memory prior to modulate the process of image synthesis. Image synthesis is central theme in a number of hierarchical models of the visual system, including interactive activation and predictive coding (McClelland & Rumelhart, 2020; Grossberg. 1987; Mumford, 1992; Rao & Ballard, 1999; Lee & Mumford, 2003), and is hypothesized to take place through the top-down feedback connections between areas in the visual cortex. These priors allow the synthesis process to look beyond the current spatial context and utilize the prototype memories learned and accumulated over time. Thus, the
Equual contribution. "Unfortunttely the data not publicly available yet.

---

Published as a conference paper at ICLR 2022
"grandmother cells" serve structural conceptual priors in memory attention process during image generation.
To emphasize the importance of having an attention mechanism that operated beyond the current image representation, we name our proposed memory-based attentional process Memory Concept Atlention (MoCA). MoCA is module that can be plugged into any layer of pre-existing generator architectures in the GAN framework. We test our model by performing extensive experiments using the state-of-the tft StyleGAN2 (Karras et al., 2020b) and a newly proposed few-shot image generator FastGAN (Liu et al., 2021). Our experiments demonstrate the utilization of prototype information accumulated within the semantic clusters during training can improve few-shot image generation on unimal-Face Dog (Si & Zhu, 2012), 100-Shot-Obant (Zhao et al., 2020), ImageNet- 100 (Russakovsky et al., 2015), COCO-300 (Lin et al., 2014), CIFARIO I0 (Krizhevsky et al., 2009) and Caltech-UCSD Birds (CUB) (Welinder et al., 2010). Additionally, we also find that generators with MoCA can resist a certain degree of injected noise corruption during testing time, suggesting that attending to structured memory prior during generation can improve the robustness of the model. Our goal here is to explore the utility of prototype cells as memory priors in a standard computer vision image generation task, with view to gaining some insights into the advantages of having these "grandmother" neurons in the visual cortex at a functional level.
2 RELATED WORK
Visual Concept Learning Visual concepts, defined as intermediate level semantic features, have been shown to be effective in overcoming misclassification of objects due to occlusion when used in feedforward voting scheme (Wang et al., 2017). Explicit representations of visual concepts using prototype neurons might also serve as effective reconfigurable parts for building compositional machines (Bienenstock et al., 1997; Geman et al., 2002; Zhu & Mumford, 2007). In this paper, explore the use of visual concepts in the form of prototype memory priors to provide temporal spatial and temporal contextual modulation with attention mechanism for the complex compositional task of image generation.
The attention mechanism (Vaswani et al., 2017) in deep learning is popular in Natural Language Processing (NLP) and is also known as "non-local networks" in the vision commu- nity (Wang et al., 2018). Zhang et al. (2019a) introduced self-attention into generative models in the Seff-Attetiion GAN. Since then, adding self-attention into Generative Adversarial Networks has be- come a standard practice. Brock et al. (2018) and Esser et al. (2020) demonstrate the benefits of using self-attention in high-fidelity image synthesis models. However, current self-attention mechanisms in GANs only utilize contextual information within the same image to modulate the activation. this work, we propose the use of memory cache of intermediate- visual conceptual prototypes to provide additional modulation in addition to the standard self-attention. In our work, the activation mechanism attends not only to its spatial context within the image itself, but also attends to a cached set of memory prototypes accumulated over time.
Self-Attention
Prototype Memory Mechanism A memory bank can capture diverse features over long time period, and has been shown to be effective in other domains. Wu et al. (2018) utilizes a memory bank in contrastive learning to obtain more diverse negative samples contrast with. Caron et al. (2020) also uses a memory queue during training to accumulate representative negative samples. He al. (2020) shows that the stability of the features accumulated in the memory bank can be improved using a momentum-updattd encoder, a strategy we will also use in learning our prototypes. SimGAN (Shrivastava et al., 2017) introduces the image pool trick which uses a buffer to store previously generated samples, in order to make the discriminator focus not only on the current training batch, but also improve itself based on memory. The major innovation of our work is the proposal, inspired by recent neurophysiological findings, that we should have memory banks at intermediate levels, and, in principle, every level of the visual hierarchy. While early memory bank work stored images at the instance level for object recognition, which is not useful for image generation, we argue that memory banks at lower levels of the hierarchy can store prototypes of parts and sub-parts which are useful for image generation. Few-shot image generation, in partieular, can benefit from memory mechanisms that support flexible composition and decomposition of parts, particularly when data are limited.
Few-Shot Prototypes Learning Few-shot learning refers to performing computer vision tasks when training data are very limited. One of the popular ideas in the few-shot learning literature is to form distinct prototypes from the training set and use them during testing (Snell et al., 2017). Although
2

---

Published as a conference paper at ICLR 2022
CNN Feature A
Memory
Max Cosine Similari
Cosine Similarity
Cosine Similarity
Cosine Similarity
Hm,(t))
Number of Prototypes (T) in Each Cluster
8(-)
MatMul
Softmax
Hyper columm Ay
MetMul
MoCA
O(-)
h
MatMul
Softmax
v-(-)
MatMul
Self-attention
Figure Attention layer using MoCA and Self-Attention. In MoCA, the input activation Akj is first transformed into low dimensional space via x convolution 0(-) and used to select its closest semantic cell in winner-take-al process. The selected semantic cell will allow the prototype memory cells in its cluster to participate in the MoCA process, generating modulation that is then mapped self-attention path,
by a Ix1 network O-)) from the embedding space back to the feature space. the entire feature map A is transformed into key and value via two corresponding 1 x convolution <(.), (-) and then attend with query vector (encoded from Ary) and then mapped back to the feature space. Finally, the outputs from two paths aggregated together to form the input to the next layer. Note that decoder (-)) and query encoder 0(-) are shared across two paths.
MoCA also forms prototypes during the training stage and uses them during inference, there are two important distinctions between our work and Snell et al. (2017): (1) Snell et al. (2017) forms prototypes at the instance level whereas MoCA's prototypes are generated at the intermediate parts level. (2) Snell et al. (2017) simply selects the closest prototype obtain discrete class prediction, MoCA, on the other hand, employs an attention process to continuously modulate the activation features based the prototypes, hence can be applied to broader set of tasks, including image synthesis, which predicts continuous pixel values.
Few-Shot Image Generation Few-shot image generation task is challenging task GANs very data-hungry and inefficient. Unconditional image generation especially difficult among the current few-shot image generation schemes. Different solutions have been developed in the literature. Recent works such as DiffAug (Zhao et al., 2020), StyleGAN-Ada (Karras et al., 2020a) propose differentiable augmentation avoid overfitting in the discriminator, InsGen (Yang al., 2021) proposes to use a contrastive learning objective to enhance the adversarial loss in the few-shot generation setting. These works mostly proposed methods improve the discriminators so that they can provide better error signals to the generator, but do not specifically address the generator architecture. Another line of research proposed generator architectures to avoid mode collapse. The state-of-the-ane architecture for few-shot image generation (hundreds of real images) is FastGAN (Liu et al., 2021). Here, we propose new architectural change on the generator side and make comparison against FastGAN architecture extensively in the experiment section.
3 METHODS
Our key contribution is the introduction of a novel prototype-basee memory modulation module improve the generator network of a GAN. The activation from the previous layer is modified by

---

Published as a conference paper at ICLR 2022
CNN Feature Map
CNN Feature Map A
MoCA Output
H
teag Bone salin
Attention
MoCA Operation
Hyper-column
Update This Cluster
dail
MoCA Operation
Hyper-oouumn
Cosine
Max Cosine
Cosine Simifarity
Momentum Updoted (-)
MoCA Operation
Kandomly Choose
Replace
MoCA Layer Operations
Memory Update Mechanism
Figure 2: Left: MoCA Layer overview. Each hyper-column the feature map processed by the MoCA Operation specified in Figure to generate modulation to modify the activation of that hyper-column before passing it onto the next layer. Right: Memory Update Mechanism. When updating the memory, a momentum-updater projection head o(.) maps the hyper-column activation vector to the prototype memory space and later incorporated into the matched semantic cluster in the memory bank using random update policy.
attention processes: contextual modulation with Memory Concept Attention (MoCA), and 2) spatial contextual modulation within the generated image itself (Self-Attention). Our module takes feature map in the GAN hierarchy as input combines the results of these two mechanisms to modulate the feature map for further downstream processing. A high-level overview of our model is shown in Figure
Formally, we denote the input to our MoCA layer to be the activation of particular layer, A R The output produces modulation H E Raxexhxx that update A into Á. To allow the information to flexibly modulated, we first transform A by x 1 convolutions into a lower- dimensional space via functions ((-), Φ()), and (x),, where {0(A), ((A), ((A} RuxE xhxw
3.1 PROTOTYPE MEMORY LEARNING
In this section, we discuss the organization of our prototype memory that is used as one route of mod- ulation. Our prototype concept memory arranged hierarchically into semantic cells and prototype cells. As shown in Figure 2, each semantic cell is cluster mean representative of cluster of prototype cells. Formally, suppose our memory P consists of M semantic cells = (K1, 1K,KK K2.... KM)- For each of the semantic cells K; there are T of prototype cells {E(+) (() E3 ,.... E stored in the
memory cell associated with K,, where E and K; both E R and K, = (E / is the mean of the stored prototype cells. These prototype cells come from the feature maps in previous iterations after being transformed via momentum updated context encoder Φ(-) and are updated in the memory at the end of every training iteration. o(-) is a momentum counterpart of o(.) and its parameter is updated as shown in Equation 1 with the momentum parameter m. Φ(-) does not change as rapidly as o(-) that the prototype learned are more stable, accumulating information beyond the current training batch. The rationale is similar to that in (He et al., 2020)).
de - 29 (1 - m) + dgg m
After transformed by ō(-) to a low-dimensional space, the activation at each hypercolumn (pixel location) in the feature map is assigned its closest semantic cluster and replaces existing prototype cell in the memory bank of that cluster. We use a random replacement policy to prevent the prototype cells from all collapsing fashion, i.c. we update K; as the cluster mean of the ith cluster of prototypes after the entire memory updated based on the recent batch.
trivial solutions. The update is done batch synchronized
4

---

Published as a conference paper at ICLR 2022
3.2 MEMORY CONCEPT ATTENTION
As we described in section 3.1, our memory is a cache of hypercolumn activation patterns gathered across the entire training set as well as from different training periods. By organizing them via semantic cells (Kis), we can route more diverse information into the final activation while still maintaining high data efficiency, hence yielding higher quality image synthesis task. In this section, present the detailed operation given existing memory P and the input feature transformation {0(A), ((A), ({A} E R x x
During the memory attention process, activation column E R from (A) first selects the closest semantic cell K2 and retrieves the associated prototype cell matrix E RexH from memory, where each column j of ( is a prototype cell
E E R belonging to the cluster of the selected semantic cell K₁. The next step is to attend to the activation column with the prototype cell matrix E(t), First, a similarity score is calculated as = [E ('a a. A nonlinear softmax normalization is then applied to s to obtain the normalized attention weight /3 where for each entry Bo (t {1, .... T)) Equation applies.
B,- exp 8,))
i= exp(s:)
Using the (softly) normalized attention weight B,, can build the retrieved information from memory hm E 3 (hm.. E RR) for the activation column By applying the same operation to the activation a at every spatial location as well as to every image in the batch, we obtain Hm, E Ruix xhxw
3.3 SPATIAL CONTEXTUAL ATTENTION
While the memory prior is important, spatial contextual information also plays a role in the activation modulation. Therefore, we also use non-local network to implement spatial contextual modulation in the same layer. Specifically, we first compute a affinity map between ((A) and of A), denoting as S = ()))"((()) Each row of S is then normalized via softmax allow rather sparse attention weight S be computed. S then multiplied with ((A) to get the spatial contextual modulation tensor H. E Raxé xhxw
3.4 INTEGRATE TWO ROUTES OF MODULATION
Finally, we integrate the retrieved information from memory Hm, and the spatial contextual mod- ulation H, by element-wise addition, H Hm ( H., which then transformed back to the original feature space via the 1 x convolution O(-). A learnable parameter y is multiplied to the multiplication as weight and add back to the input activation, ie. À = (H(H) + A, which is then output into the next layer in the generator.
4 EXPERIMENTS
The proposed MoCA module is a generator architecture improvement and is not specific to the discriminator's 's training techniques. Note that some the existing works on unconditional few-shot image generation (DiffAug (Zhao et al., 2020), ADA (Karras et al., 2020a), etc.) improve the quality of image synthesis by focusing on the discriminator side. For example, DiffAug proposes to perform augmentation of the generated image before being fed into the discriminator to prevent discriminator's overfitting. Although this line of work is valuable, is orthogonal and complementary to MoCA, as MoCA is generator architecture that can be trained under all of these discriminator training techniques. In the following experiments, we use MoCA enhanced generator with either DiffAug or ADA augmented discriminators.
Datasets and Metrics We validate the performance improvements of adding the MoCA module for few-shot image synthesis with 6 datasets. Namely, we perform comparisons between the baseline architecture and the MoCA-enhanced architecture on Animal-Face Dog (Si & Zhu, 2012). 100- Shot-Obama (Zhao et al., 2020), ImageNet- 100 (Russakovsky et al., 2015), COCO-300 (Lin et al., 2014), CI (Krizhevsky et al., 2009) and Maltech-UCSD Birds (CUB) (Welinder et al., 2010). Animal-Face Dog consists of 389 dog images while Sa--ShhtOOaaat contains 100 Obama faces with different expressions. We use 256x256 resolution for both of these datasets across all experiments. To further challenge the few-shot image generation task, we select 100 and 300 images from one class of ImageNet ("Jeep" category) and MSCOCO ("Train" category) respectively to compose
5

---

Published as a conference paper at ICLR 2022
Figure 3: Generated images from MoCA different datasets. Different resolutions of images considered. The biggest one is 256x256, the middle one is 64x64 and the smallest one on the right is from CIFAR-10 with 32x32 resolution.
the ImageNet- 100 and COCO-300 datasets (the details of the selected image list can be found in the Appendix 8.5). In addition, also conduct experiments on CUB (5,990 256x256 resolution wild bird images) and CIFARIO datasets (60,000 32x32 images span across 10 categories). For evaluation metries, we use a widely adopted metric. Fréchet Inception Distance (FID) (Heusel et al., 2018) to measure the quality of generated images. For most of the experiments, except the one for the evaluation of model robustness against noise, the standard deviation of FID are often less than 1% relatively and much smaller than the performance improvement. To strengthen confidence in the evaluation of the image synthesis performance, we also evaluate the Kernel Inception Distance (KID) metric (Binkowski et al., 2018), which is found to be more descriptive for the few-shot image generation tasks.
Baselines To demonstrate that MoCA is not limited to certain backbone architecture, we add MoCA to two state-of-the-ar architectures, FastGAN (Liu et al., 2021) and StyleGAN2 (Karras et al., 2020b), Of note, FastGAN is a state-of-the-art architecture that is specifically targeted on image synthesis in the extremely low data regime. It is an important baseline architecture that provides reasonable few-shot synthesis given only hundreds of real images and as few as 15 GPU hours of training time. Additionally, we also add MoCA to StyleGAN2, which is a more powerful and generic model but requires more data and computing resources. We will showcase that the McCA layer can be useful for generators in both architectures Section 4.1. The implementations of the baseline models are based on their official publicly available Py PyTorch implementations. We follow the best training configuration reported in the original papers to train our models.
4.1 FEW-SHOT IMAGE SYNTHESIS PERFORMANCE
Here, we present the results of the experiments on 4 small datasets, each of which has only hundreds of real images (Animal-Face Dog (Si & Zhu, 2012), 100-Shot-Obama (Zhao et al., 2020), ImageNet- 100 (Russakovsky et al., 2015), and COCO-300 (Lin al., 2014)). Qualitative results are shown in Figure 3. Meanwhile, we show quantitative comparisons between the best performance of MoCA and baseline models for each dataset in Table (FastGAN) and Table 2 (StyleGAN2). Note that the KID is shown on the scale of 10 As shown in the results, we observe consistent improvements in terms of image synthesis quality in this difficult low data regime when adding MoCA to the generator. In terms of FID score, we observe that, using the FastGAN base architecture, MoCA can bring a 5.8% improvement on Animal Face Dog, 13.8% improvement Obama, 21.7% improvement - and 12.4% improvement on the COCO-300 dataset (Table 1). Similar improvements are also observed with KID metrics.
To further test MoCA's enhancement on more powerful model, we run experiments with a StyleGAN2 base architecture. Using the FID evaluation metric, we observe a 5.1 improvement on Animal Face Dog dataset, 8.1% improvement on Obama dataset, 14.1 %% improvement on ImageNet- 100 dataset and 17.3% improvement on COCO-300 dataset. Note that ImageNet-100 and COCO-300 dataset are very challenging as they contain much more diverse scenes than Animal Face Dog and Obama. To compensate for the model size of the StyleGAN2 base architecture, we downsize ImageNet- and COCO-300 to 64x64 image size when training using StyleGAN2 and MoCA-StyleGANY From the results, we observe that MoCA- MoCA--StleGAN2 consistently synthesizes better images compared to the baseline despite the image size. Additionally, we also observe that different augmentation methods

---

Published as a conference paper at ICLR 2022
Animal Face Dog
Obama
ImageNet- 100
COCO-300
Generator Architecture
Discriminator Training
FID J
KID
FID 4
KID 4
FID 1
KID 4
FID 1.
KID 1
FastGAN
DiffAug DiffAug
51.25
17.11
43.13
13.52
66.44
22.31
30.43
6.02
McCA-FasGANN (ours)
48.27
13.83
37.19
7.81
52.01
7.23
26.67
4.53
Table 1:
Quantitative results of MoCA on the FastGAN- architecture. FastGAN (Liu et al., 2021) is trained following the setting in original paper, which uses differentiable augmentation (DiffAug) (Zhao et al., 2020) to train the discriminator. All experiments in this table conducted using 256x256 size images. Results suggest a consistent improvement from the MoCA layer across different few-shot generation datasets.
Animal Face Dog
Ohama
ImageNet- 100
COCO-300*
Generator Architecture
Discriminator Training
FID J
KID 1
FID J
KID
4
KID
FID
KID
StyleGAN2
ADA ADA
58.30
22.72
28.22
5.44
$4.32
4.53
77.81
38.66
MoCA-StyleGAN2 (ours)
55.35
17.62
25.93
4.91
46.71
3.03
64.32
20.53
Table 2: Quantitative results of MoCA on the StyleGAN2 base architecture. Following the setting in StyleGAN2- ADA's original paper (Karras et al., 2020b,a), we use adaptive discriminator augmentation (ADA) to train the discriminator. We use 64x64 image size when performing experiments StyleGAN2 and MoCA-StyleGAN on ImageNet- 100* and COCO-300* and 256x256 image size synthesis Animal Face Dog and Obama dataset. Consistent improvements when adding McCA layer are observed despite the small image size and datasets.
CIFAR-10
Animal Face Dog
Ohama
ImageNet- 100
COCO-300
Generator Architecture
Generator Architecture
FID J
KID L
FID
KID
FID 1
KID J
FID
KID
FID 1 KID J
StyleGAN
5.19
2.43
FauGAN
31.25
17.11
43.13
13.52
66.44
22.31
30.43
6.02
SA StyleGANN
5.60
2.79
SA-FasGGAN
51.17
16.57
38.93
9.37
56.69
16.93
29.66
6,31
McCA-StyleGANN (ouns)
1.39
McCA-FasGAN (ours)
37.19
781
52.01
26.67
Table 3: Ablation study on the improvement of McCA comparing with GAN employing standard self-
Attention (Zhang et al., 2019a)
on the discriminator side do not affect MoCA's improvement, as FastGAN based models use DiffAug whereas StyleGAN2 based models employ the ADA technique.
4.2 ABLATION STUDIES
In this section, we present ablation studies for MoCA layer compared with self-attention alone and the importance of the momentum head design. Additional ablation studies on the design of memory clustering organization used in MoCA can be found in Appendix 8. and more visualization analysis can be found in Appendix 8.6.
Self-Attention v.s. MoCA On both StyleGAN2 and FastGAN, we compare the performance of MoCA with a standard self-attention module verify that the performance improvement of our model is not only due to the self-attention block, as several previous works have highlighted the advantage of applying self-attention on GANs to capture non-local dependency (Zhang al., 2019b; Brock et al., 2018). Table 3 shows that in spite of the baseline architecture, MoCA consistently outperforms the standard Self-Attention module, suggesting that MoCA module plays an irreplaceable role in our method.
Importance of Momentum Update Mechanism We also study the effect of different ways to update the concept encoder. Table shows that the models' FID on multiple datasets generally improves when using momentum update the concept encoder. With momentum, the FID of FastGAN on larger and diverse datasets like AnimalFace Dog and CUB is notably better. This supports our motivation for using a momentum-updaated encoder to build a more generalized memory bank that contains broader information beyond the current training batch.
FID
Model
AnimalFace Dog
Obama
CUB
MoCA with momentum
48.27
37.19 46.25
25.66
McCA w/o momentum
63.08
51.74
Table 4: Ablation study FastGAN + MoCA with different update mechanism
4.3 PROTOTYPE CONCEPT ANALYSIS
Here, we demonstrate that MoCA learn in distinct semantic concepts in unsupervised manner, represented by the "grandmother cells" or their representatives, during adversary learning for image synthesis.
7

---

Published as a conference paper at ICLR 2022
Original
Cluster 0
Cluster 2
Cluster
Cluster
Cluster 16
Cluster 17
installed and Figure 4: Visualizing Cluster Semantics in MoCA. We use MoCA-FastGAN model trained MSCOCO-300 dataset for visualization (Lin et al., 2014). Each row is a generated image. We compute the cluster assignment for each hypercolumn in the layer where MoCA highlight the receptive fields of hypercolumns with white bounding boxes. We further group the visualization of the receptive fields by clusters (columns). For example, for column "Cluster 0", all white bounding boxes are the receptive fields of the hyper-columns that are modulated via Cluster O's prototypes. We observe that different clusters have different semantics since their prototypes tends to modulate different semantic regions of the images (See discussion in Section 4.3).
Semantic Concepts in MoCA We assigned a semantic cluster to each hypercolumn the 16 16 feature map during feedforward generation pass (cluster selection mechanism is described in Method Section 3.2). We label the receptive fields or projection fields of all the hypercolumns associated with each chosen prototype semantic cluster as white bounding boxes the generated image, with some examples shown in Figure 4. We can observe that different clusters encode distinct semantic concepts. For examples, Cluster 0 tends be associated with train rails. Cluster covers area of uniform color such as sky and ground. Cluster 17 pays attention to the side of the train. Cluster focus on the trains themselves, particularly the front. Along each row, we observe that the influence of MoCA for specific image can be decomposed into different clusters. Some clusters are more popular (e.g. Cluster 0, 2, 8) while other clusters are used more selectively (for example, cluster 17 for the third image).
Understanding Prototype Cells To understand the prototype cells that are cached inside each cluster of MoCA, we identified the generated image patch in the receptive field of any hypercolumn whose activation is closest to the prototype cell's memory in the assigned cluster of that hypercolumn from 100 image generations. Figure show that the image patches closest to prototype memory are similar visually, and that prototypes belonging to the same cluster are semantically related but distinct in visual features. We can see that different prototype cells are specialized sub-parts of the semantic clusters they belong to. For example Cluster 0, prototype 20 is cached activation for rail or train tracks, but other prototypes include the top of the trains. Similar observations also be made for Cluster and (Figure 5). These three clusters are queried frequently and therefore likely contain diverse but semantically related prototype memories. The prototype memories resemble the "visual concepts" of (Wang et al., 2017) and can potentially be used as visual concepts to implement hierarchical compositional system for image synthesis.
8

---

Published as a conference paper at ICLR 2022
Cluster Proto 20
Cluster Proto 103
Cluster Proto 265
Cluster Proto 269
Cluster Proto 328
Proto
Cluster Proto
Cluster Proto 264
Figure 5: Visualizing different prototypes. Each image patch shown above is cropped based on the receptive field of hyper-column. For each prototype inside MoCA's memory (each row above), we find hyper-columns whose activation will be largely modified from that prototype ("largely" if they are similar) during attention process and crop their corresponding receptive field in the generated images based on the convolution architecture (Details refer the Appendix).
4.4 ROBUSTNESS AGAINST NOISE
We evaluate the noise robustness of our proposed memory concept attention module and compare it against the standard self-attention module. The evaluation conducted on FastGAN models trained without any injection of noise into the intermediate layers. In the evaluation stage, inject Gaussian noise of different magnitudes with variance from 0 to into the input feature map of the MoCA layer and standard self-attention layer. Our results (Table show that, without the help of external memory, self-attention is more vulnerable to the noise attack, while MoCA is generally less sensitive noise. We suspect that with MoCA, the feature map can attend previously-stored concepts in the memory bank to retrieve frese-free part information, which alleviates the influence of noise perturbation. We also find that, no-cluster MoCA (All prototypes belong to the same cluster and jointly inference every activation Asj. i.e. M=1) is more robust under higher levels of noise. We hypothesize that the feature map can attend to more concepts in no-cluster MoCA, increasing the chances of obtaining correct bias.
Noise
Model
0.7
0.5
0.3
BD Boüse
Self-attention
163.71 (17
114.17
(255100)
75.48 (1:4552
45.42
(13.61)
26.65 (:10.33)
McCA (isl cluster)
+9.71(=2.79)
63,84 (#:1.68)
$4.63 (+:1
45.82
(+:1228) 1.28)
37.04 (+H0.31)
MoCA (with chostr))
117.38 (2:4.32)
(#:4-0)
59.24 (:22
38.93 (1:3711
23.98 (:10.12)
Table Image synthesis FID by MoCA-FastGAN on CUB under different levels of noise injection.
5 CONCLUSION
In this paper, we introduced a module called MoCA, which can be inserted into any layer of hierarchical neural network such GAN improve image synthesis. MoCA caches part prototype memories over time, dynamically updated using a momentum encoder, and has the ability to modulate the continuous response of intermediate layers via an attention mechanism. In contrast to earlier memory bank studies which stored representation of the entire images as prior in the latent space for object recognition, the prototypes MoCA are remembered parts and sub-parts objects, that can be extracted at every level of the hierarchy, and flexibly composed for image synthesis. Our work was inspired by the discovery of a diverse set of highly selective complex feature detectors in VI, which correspond the prototype cells in MoCA. The selection of different prototype clusters via semantic cells could correspond to switching circuits mediated by inhibitory neurons. While we do not claim that there is any serious correspondence between model and neural mechanisms circuits, the effectiveness of MoCA might still provide some insights to the roles of these "grandmother neurons" functional level.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This study investigates novel proposal for understanding the computational advantages having prototype-like "grandmother" neurons the visual cortex, namely, that these prototype neurons can serve as auxiliary memory priors to modify both the embedding space of the self-attention mechanism as well the activation patterns of neurons during the image synthesis process. While the motivation is for advancing our understanding of contextual modulation in neural systems, our work also results a novel mechanism contextual modulation in deep learning systems, particularly in image generation, and thus has broader positive impact machine learning, particularly on few shot learning. However, image generation technology can be misused for misinformation, with adverse social impact, c.g. the creation of "deepfakes. On the other hand, advancing understanding of image generation also crucial for combating deepfakes and other misuses of image generation technology by bad actors.
REPRODUCIBILITY STATEMENT
Our project code can be found htttpp////gttha...cc///aaaaaaa/p
ACKNOWLEDGEMENT
This work supported by an NSF grant CISE RI 1816568 awarded to T.S.L. This work is also partially supported by the start-up fund provided by CMU Mechanical Engineering Department.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Elie Bienenstock, Stuart Geman, and Daniel Potter. Compositionality. mdl priors, and object recognition. Advances in neural information processing systems, pp. 838-844, 1997.
Mikolaj Binikowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1801,00 01401, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1009, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:20/2.0984/ 2020.
Stuart Geman, Daniel Potter, and Zhiyi Chi. Composition systems. Quarterly of Applied Mathe- matics, 60(4):707-736, 2002.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint vrXiv:1406.200 2014.
Stephen Grossberg. Competitive learning: From interactive activation adaptive resonance. Cogni- live science, 11(1):23-63, 1987.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium, 2018.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119 119,,199,899 2020b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tai Sing Lee and David Mumford. Hierarchical bayesian inference in the visual cortex. JOSA A, 20 (7):1434-1448, 2003.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. arXiv e-prints, pp. arXiv-2101, 2021.
Xudong Mao, Qing Li, Haoran Xic, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks, 2017.
James L McClelland and David E Rumelhart. Distributed memory and the representation of gen- eral and specific information. In Connectionist psychology: text with readings, pp. 75-106. Psychology Press, 2020.
11

---

Published as a conference paper at ICLR 2022
David Mumford. On the computational architecture of the neocortex. Biological cyberneties, 66(3): 241-251, 1992.
R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 13507045):1002-11000 2005,
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87) 1999.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211-255 2015.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2107-2116 16, 2017.
Zhangzhang Si and Song-Chun Zhu. Learning hybrid image templates (hit) by information projection. IEEE Transactions on Patten Analysis and Machine Intelligence, 34(7):1354-136) 2012. doi: 10.1 109/TPAMI 1.227.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv: arXiv:1703.000 100..07737.773 2017.
Shiming Tang, Tai Sing Lee, Ming Li, Yimeng Zhang, Liu Fang Xu, Y, Teo Benjamin, and Hongfei Jiang. Complex pattern selectivity in macaque primary visual cortex revealed by large-scale two-photon imaging. Current Biology, 28:38-48, 2018a.
Shiming Tang, Yimeng Zhang, Zhihao Li, Ming Li, Fang Liu, Hongfei Jiang, and Tai Sing Lee. Large-scale two-photon imaging revealed super-sparse population codes in the v1 superficial layer of awake monkeys. elife, 2018b.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 986)) 2605, 2008. URL http: //tpp:://///////////////epepea http://tl......eeeeeeeeem wttppppprr////g////./.
vandermaaten html.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 0at::1706037762, 2017.
C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The Caltech-UCSD Birds-200-241 Dataset. Technical report, 2011.
Jianyu Wang, Zhishuai Zhang, Cihang Xie, Yuyin Zhou, Vittal Premachandran, Jun Zhu, Lingxi Xie, and Alan Yuille. Visual concepts and compositional voting, 2017.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report
Califoria Institute of Technology, 2010.
Zhirong WU, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. arXiv preprint arXiv: 009780000007...00 2018.
Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Data-efficient instance generation from
instance discrimination. arXiv preprint arXiv:2 106.04666, 2021.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7364-7363. PMLR, 2019a.
12

---

Published as a conference paper at ICLR 2022
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks, 2019b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training, 2020.
Song-Chun Zhu and David Mumford. A stochastic grammar of images. Now Publishers Inc, 2007.
13

---

Published as a conference paper at ICLR 2022
8 APPENDIX
8.1
ABLATION: IMPORTANCE OF MEMORY CLUSTERING ORGANIZATION
The second aspect we study the impact of the clus- tering mechanism on the overall model performance. We find that for MoCA, single concept pool with sufficiently large size, can still has similar perfor- mance compared against MoCA with multiple clus- ters.
# Cluster
Size of Cluster
FID +
KID 4
512
5.30
2.53
8192
4.76
1.52
256
5.10
1.96
As shown in Table 6, the performance of no- cluster MoCA containing 8192 concepts inside, is at the similar level MoCA with 32 clusters. We hy- pothesize this is because, although there is no cluster- ing mechanism enforce the memory bank learning diverse concepts in no-cluster MoCA, given a very large pool of concepts, it is still able to memorize diverse and useful concepts. Despite the finding that no-cluster MoCA can have similar performance to cluster MoCA with a sufficiently large pool, we argue that using the clustering mechanism is still superior. Uusing the clustering mechanism greatly reduces the amount of vectors involved in the attention calculation, which enables us to build a larger memory bank for complex datasets.
20
256
4.91
1.61
20
1024
4.90
1.56
256
4.68
1.39
Table 6: StyleGAN2 on CIFAR-10 with dif- ferent architectures. The first column denotes the number of cluster in the memory, 1 in- dicates no clustering mechanism is applied. The second column denotes the number of concept vectors attributed single cluster.
8.2 MOCA ON LS-GAN
To strengthen conclusion MoCA, have also conducted experiments the least-square GAN (LS-GAN)(Mao et al., 2017) framework which has a different loss and architecture compared FastGAN and StyleGAN, We found MoCA can still provide a certain amount of improvement, consolidating the advantages of the MoCA layer.
Animal Face Dog
Obama
Cifar-10
Generator Architecture
Discriminator Augmentation
FID
KID 4.
FID 4.
KID
FID
KID
LS-GAN MoCA-LS-GAN (ours)
DiffAug DiffAug
135.83
109.32
180.35
226.45
56.37
4.21
122.89
91.14
172.97
206.38
50.48
3.86
Table 7: Quantitative results of MoCA LS-GAN based architecture. All experiements in this table are conducted using 64x64 size images. Results suggest a consistent improvement of MoCA layer.
8.3 LIMITATION AND FAILURE
Despite MoCA can bring in consistent improvement on most of the dataset with limited data, we also observe case where MoCA is less effective. We found that adding MoCA on StyleGAN for training Grumpy-cat leads to slight performance decrease and leads to small improvement on FastGAN. This suggests that when the underlying dataset is less diverse and the base network quite large, MoCA cached concepts can potentially cause more distraction during generation, leading to performance setbacks.
Grumpy
Generator Architecture
Discriminator Augmentation
FID J
KID J
FasIGAN
DiffAug Dif[Aug
26.65
5.71
McCA-FastGAN (ours)
25.68
5.18
StyleGAN2
ADA
24.17
3.82
McCA-StyleGAN2 (ours)
ADA
24.65
3.98
LS-GAN McCA-LS-GAN (ours)
DiffAug DiffAug
99.92
88.43
83.82
73.22
Table 8: Quantitative results of MoCA on Grumpy-cat dataset.
14

---

Published as a conference paper at ICLR 2022
8.4 ALGORITHM
Here, we present the algorithm that is presented in Figure 1 and Figure 2 in the main text.
Algorithm 1: MoCA Self-Attention in the Generator
Result: Updated activation Á after MoCA and Self-Attention Updated Memory P. E Ruxexhxw <- from previous layer;
{0 (O(A), ((A), ((A) & I xhxw w t- transform into č dimensional space via Ixl conv;
Memory Concept Attention:
for every spatial column R in ((A) do
Choose prototype semantic cell K, where i i argmin,||KK; all::
E R + Retrieve Prototype Component Cells for K; from memory;
E E
compute dot product with memory;
3 Σ empfa)
expler
hm
E 3
E
# hm Re:
end
H &- Combining hm
# Hm Raxxxhxw
# Attenint;n;
S + [0A)]* ((A);
$ soflmax(S):
H < (
# HH, E Raxxxhxw;
H Hm H,;
À = (H( + A;
# Memory Update;
Am - -(A::
for every column am E R in Am do
Choose prototype semantic cell K; where argmin, ||;; am
Randomly update one column in O with am; end
09 (1 - m) + dg
# momentumly update the Ø
8.5 EXPERIMENTAL DETAILS
In this section we provide details for how we implement MoCA on FastGAN and StyleGAN respectively. For reference, we provide our code and image ID list of MSCOCO, ImageNet subset used in the experiments at https: //tp://gittt com/Crazy datasets used in experiments are publicly available and for non-commercial usage only. Among them, CIFAR-10" 10 is attributed to Krizhevsky (2009), Obama, Animal Face, Grumpy Cat are attributed to Zhao et al. (2020), and Caltech- Birds-200-2011 (CUB) is attributed to Wah et al. (2011).
ceceCce_eeeeeeeeeeeeeeeee_eeee All the
We insert MoCA layer between the last but one residual block and the last residual block of the generator. Given the input feature map A the last but one residual block f(1-1), the last residual block f() and MoCA module 4, suppose the original image generation procedure on the last two blocks is:
= (AL)
(3)
Then the feedforward procedure including MoCA can be defined as:
x (+(F(-)(A)))
(4)
hnttps://www.d http:///........ceee//////eaaaaa http:://wwwwrrtoooaccccccceeeeps 1.hmll
15

---

Published as a conference paper at ICLR 2022
where & is the generated image. Please note that, to enable that MoCA can be flexibly integrate into arbitrary layer, MoCA will not change the channel numbers and resolution the feature map.
8.5.1 FASTGAN
Network Architecture and Optimization We use exactly the Fast-GAN architecture Liu al. (2021) as the backbone. MoCA layer is inserted after the 64x64 resolution layer for all experiments run for CUB, AnimalFace Dog, Obama and Grumpy-cat. We use Adam optimizer for training and 0.0002 for learning rate with $1 0.5 and B2 0.999. Differentialbe augmentation is used during training. We utilize the loss proposed in (Liu et al., 2021) for stable and optimal training. For all our experiments, 100K iteration is run perform fair comparsion.
Computational Resource With FastGAN and MoCA, it takes about 15 hours to train 100K itera- tion on a single GTX-1080Ti GPU.
8.5.2 STYLEGAN 2
Network Architecture and Optimization We use exactly same architecture from Karras al. (2020a) as the backbone. We use Adam (Kingma & Ba, 2017) optimizer with learning rate of 0.0025 and 31 0, Bo 0.99. We adopt the same training configuration as the Official PyTorch implementation of StyleGAN 2 (including Adaptive discriminator augmentation, path length regularization, lazy regularization (Karras et al., 2020b), and non-saturating logistic loss (Goodfellow et al., 2014)). For the training on the large dataset (C))AR-10), we train models for 200k iterations of gradient updates, and 100k iterations for few-shot datasets.
Computational Resource We train StyleGAN2-based model with 4 RTX-2080Ti GPUs each run. The training of each model takes roughly 20 35 hours to complete, depending on the image resolution.
8.6
MEMORY CONCEPT ANALYSIS
map
Memory Concept Attention Module
[ch,
Prototypes [ch, M)
Key
Key
transposc
Imnopooe
[h w]
2
which prototype has the largest affinity?
pick most
M
frequent indexes
(M hu
Index of closest prototypes
[h, w]
Affinity prototypes
query the i-th channel
binary mask decomposition wirdt. top-3 clusters
affinity heatmap prototype
Figure 6: Schematic of affinity map visualization procedure
We generate 50,000 images from random noise for CIFAR-10, and 1,000 images for 100-shot Obama. For each feature map and specific concept cluster i, count how many pixels' closest cluster (measured by the magnitude of the dot product) is i, and repeat this process for all clusters. Then sum the count and pick top-K clusters which have the most pixels, and visualize their affinity
16

---

Published as a conference paper at ICLR 2022
heatmap. To interpret the vector embeddings (concepts) stored in each concept cluster, perform three kinds of visualization: affinity heatmap w.r.t. different concept clusters, images' binary mask w.r.t. different concept clusters, and t-SNE visualization (van der Maaten & Hinton, 2008) of the feature map before and after attending with memory concept. The detailed information retrieval process is shown in Figure 6. We use StyleGAN2+MoCA's results for visualization, they perform the best on most of the dataset.
Semantic concepts In Figure 7a and Figure 7b, we show the resulting affinity maps computed as discussed above (detailed schematic of visualization procedure is shown in Figure 6). Note that we select clusters and images for demonstration purpose. Results on CIFARIO using MoCA-StyleGAN (32x32 resolution) and Obama dataset using MoCA-FastGAN (256x256 resolution) are acquired. We find that these affinity maps demonstrate that concepts formed during MoCA training cover crucial and diverse patterns for image synthesis. As shown in Figure 7a and Figure 7b, patterns of different clusters usually semantic meaningful and very distinct. On CIFAR-10, we notice that cluster 11 stores concepts related to the sky, while cluster 13 stores concepts related to the frame of trucks. Cluster 7 favors the white space while cluster 18 has "crush" on animals' body and head. Similarly, on the other dataset, different clusters also focus on very different patterns. For instance, on 100-shot Obama, cluster stores the concepts closely related to ties, and cluster contains concepts related to the light reflection on the face.
Claster
(a) CIFAR-10
(b) 100-shot Obama
Figure 7: Affinity heat-map w.r.t. different concept clusters. White: strongest affinity; Dark red:
zero affinity. Left (a): McCA- SoCA-StyleGAN2 on CIFAR10. Right (b): McCA-FastGaa on 100-shot Obama dataset. We can observe that Different cluster associated with different semantic meanings although not trained with category supervision.
Prototype attention We study the internal dynamics of MoCA by using t-SNE to visualize the changing trend of feature map before and after attending with concepts in the memory. We generate 10,000 images from random noise CIFAR-10 and extract the feature maps before and after MoCA layer. Figure 9a shows that the activation patterns (blue dots) from previous layer, after the 0 transform, are effectively influenced by MoCA (green box in Figure 1) into representation (red dots) that are pulled towards or stretched by certain familiar prototype exemplars (green dots).
Image synthesis concepts assembling By visualizing the affinity to the prototype of each pixel in the feature map, we acquire high-level understanding of what concepts different clusters are trying to capture. We can also further investigate the relationship between different concepts and the image synthesis process. We decompose the image into different binary masks with regard to top-3 clusters. The decomposition is shown in Figure 8. Concretely, given specific cluster Cj and a feature map F, we generate the binary mask w.r.t. Cs by querying whether pixels' (Fiz) closest cluster (highest affinity) is Ck or not, where white denotes true and black denotes false. We pick the top-3 clusters for each image and visualize their binary mask as shown in Figure 8. For most images, concepts from the top-2 clusters are usually related foreground and background information, while the third cluster contains concepts related to high-frequency details in an image. This demonstrate that, with MoCA, image synthesis can also be viewed as a process of retrieving different concepts from memory and assembling them together to create a realistic image.
Implicit influence of MoCA the attention mechanism Interestingly, we also observe that having the MoCA module can in turn sharpen the functional activities of the horizontal interactions. To visualize the functional connectivity of the horizontal connections, we plot the normalized rank-order
17

---

Published as a conference paper at ICLR 2022
affinity score of the self-attention map in the 16x16 resolution layer of the converged StyleGAN2 trained on CIFARIO.
Tep- actiuned Concept Cuatar
Sep actinded Concupt Chalee
MoCA til--ittrrioon
C4
(a) I-SNE visualization of MoCA dynamics
(b) Sharpening of the Context Attention Map by MoCA
Figure 8: CIFAR-10 10 image binary decomposition w.r.t. their top-3 activated clusters.
Figure 9: Modification of the activation representation and Context Attention Map by MoCA.
In Figure 9b, the purple curve represents the rank-ordered curve without installing the MoCA and the red represents that with MoCA. Clearly, MoCA sharpens the functional connectivity among the horizontal connections, even though we do not explicitly modify the self-attention process in the MoCA layer, indicating that our prototype memory is implicitly influencing the horizontal connections.
8.7 METRIC EVALUATION
For all models, we evaluate metric using the last 3 snapshots of models (with an interval of 10k gradient updates) and report the best one. We train all StyleGAN-based models for 200k iterations and all FastGAN- based models for 100k iterations. To measure the model performance on Animal-Facc Dog (Si & Zhu, 2012), 100-Shot- Obaam- (Zhao et al., 2020), ImageNet- 100 (Russakovsky et al., 2015), COCO-300 (Lin al., 2014), we sample 3, 000 noise vectors from standard Gaussian and calculate FID and KID score of the generated images. On CIFARIO 10 (Krizhevsky et al., 2009) and Caltech-UCSD (Welinder et al., 2010), we sample 50, 000 respectively to adjust for number of training data.
8.8 GENERATED IMAGES AND THEIR NEAREST NEIGHBOR IN THE DATASET
Although MoCA proposes a memory module, our improved results are not coming from trivially memorizing the training images. First, the metrics we are using (FID and KID) scores) take the generation diversity in to consideration. We sample over 3,000 or 50,000 images during the evaluation process, however the entire training dataset only contains hundreds. Second, to qualitatively test if MoCA only perform generation by memorizing the training images, we find the nearest neighbor of generated images based on the Learned Perceptual Image Patch Similarity metric (LPIPS) (Zhang et al., 20.8). The feature extractor used here is a pretrained VGG16 (Simonyan & Zisserman, 2015). In Figure 10, see that on the left is some images randomly chosen from the MoCA-FastGAN generation, and on the right is top-3 closest training images based on the LPIPS (Zhang et al., 2018) similarity metrics. We can clearly observe that the generated images are different from the training images. And more interestingly, we can see that the generated images indeed incorporate some parts level information and combine those parts level examples to compose new images. For example, the generated train example on the second row (left) may have the similar train head as the second training image, but resembles a similar train side view (the shape) as the first training image. The overall direction of the rails also have looks similar the first image but it replace the rain shelter on the boarding platform with an open sky with some electric cables hanging over. This further consolidates with the observation made in Section 4.3 that MoCA works at modulating the parts level generations. Overall, the results in Figure 10 suggests MoCA's generations are not only memorization of the training images but rather combine different parts compose new instance.
18

---

Published as a conference paper at ICLR 2022
Generated image
Top Nearest Neighbors in the dataset
Figure 10: Some randomly generated images and their corresponding nearest neighbor in the dataset. Left: Generated images from MoCA-FastGAN models. Right: Its top-3 nearest neighbors in the training dataset with the similarity score rank from left to right high low. The similarity is measured by perceptual distance (Zhang et al., 2018)
8.9 VISUALIZATION DETAILS
Details Receptive Field Localization order to generate the receptive field of each hyper- column (white bounding box in Figure 4), calculate the receptive field according to the network architectures (Liu et al., 2021). The final image is of 256 256 resolution. For a single hyper-column in the layer install MoCA (layer 16 16), there are up-sampling layers (each with factor of 2) and 7 convolution layers (each has 3x3 kernel size, stride and padding). resulting in a 66 x 66 receptive field size. The location is traced based the recursive derivation of the convolution and up-sampling operations (Goodfellow et al., 2016).
Details of Visualizing the Prototypes Since our prototypes only influence the activation via attention, it is not straightforward to visualize prototype. However, since our prototypes are used to modulate the hyper-column weighted by the normalized similarity between the 0f 0(Aij) and the prototype vector, can consider hyper-columns that have high cosine similarity with the prototype vector the projected space as an approximation of that prototype because if they are similar, there will be large weight during attention process. Since each hyper-column would correspond to patch generation in the image, we can visualize the corresponding patch and get a sense of what the individual prototype cell is representing for.
19


Reviews:
[{"review_id": "W_KCxtC7qHc", "reviewer": "Reviewer_9zZN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Novelty. A novel prototype-based layer which significantly improves the respective augmented image generation models: FastGAN, StyleGAN2.\n- Significance. The few-shot image generation is an important research topic.\n- Relevance. Will be of interests to ICLR community, especially the vision researchers.\n- Experimental Design. The performance improvements of adding MoCA layer to  two state-of-the-art architectures, FastGAN and StyleGAN2 were validated with 6 datasets of few-shot image synthesis: Animal-Face Dog, 100-Shot-Obama, ImageNet-100, COCO-300, CIFAR10 and Caltech-UCSD Birds (CUB). \n- Experimental Results. The consistent and somewhat significant improvements over the two baselines are impressive.\n- Suitable ABLATION STUDIES were done.\n- Analysis using visualization were performed to better understand the model.\n- Writing is clear and well-structured.\n\nWeaknesses\n- Minor. Only 2 baselines were used to validate the novel design. The paper will be significantly strengthen with experiments on more baseline models.\n- Minor. The modeling of the proposed method: MOCA has very limited correspondence in neuroscience. This is despite the paper's claim of being \"inspired by neuroscience discoveries\". As paper did not claim to be based on neuroscience, this is not a major concern.", "summary_of_the_paper": "Paper proposes a novel prototype-based memory modulation layer (MoCA) to improve the generator network of a GAN. The target problem is few-shot image generation. Memory is arranged hierarchically into prototype semantic cells and prototype component cells. This design is loosely inspired by the recent discovery of \"grandmother cells\" in V1.\n\nExperimental results show that in terms of FID score, using FastGAN base architecture, MoCA can bring 5.8% improvement on Animal Face Dog, 13.8% improvement on Obama, 21.7% improvement on ImageNet-100 and 12.4% improvement on COCO-300 dataset when using FastGAN as the baseline models. With StyleGAN2 base architecture, there was an 5.1% improvement on Animal Face Dog dataset, 8.1% improvement on Obama dataset, 14.1% improvement on ImageNet-100 dataset and 17.3% improvement on COCO-300 dataset.", "summary_of_the_review": "A well-written paper with novel and significant contributions to an important research topic. The design is clearly explained and the experimental results are excellent. While there are a couple of minor issues, the overall quality is sufficiently high for an acceptance by ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "If1_EVdLewa", "reviewer": "Reviewer_JpF6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a timely and interesting neural network memory mechanism, MoCA. The paper is fairly clear and well-written. However, the connections to neuroscience are weak (contra the introduction and discussion), and the image generation results raise some further questions that should probably be answered before recommending acceptance.\n\nThe authors’ central claim is that the proposed memory mechanism mechanism, which enables the network to memorize (transformed) activation vectors from images seen during training, similar to “grandmother cells” in human cortex, can enhance images generated by GANs. The connection to neuroscience here is tenuous at best — the authors draw inspiration from neuroscience, but I’m highly skeptical that the results are actually garnering any neuroscientific insight (to do that would probably require comparing these models to some form of brain recordings, but it is unclear how exactly that would be achieved with the current image-generation setup). So for the purpose of this review, I’m going to assume that this is simply a computer vision paper.\n\nQuantitatively, the authors’ efforts seems successful: GANs trained using MoCA achieve better performance on few-shot learning than identical networks without MoCA. However, it is not clear that this comparison is entirely fair: do the StyleGAN2 and MoCA-StyleGAN2 (or FastGAN and MoCA-FastGAN) networks have the same number of parameters? This is not stated, but it seems like the MoCA layers will simply increase the parameter count, potentially inflating performance without revealing a real insight.\n\nRelatedly, it seems important for the authors to test whether the MoCA layers are enabling the GANs to simply memorize the training images. In the example image figure (Figure 3), the authors should show the most similar image from the training set alongside each generated image.\n\nThe authors efforts to investigate the information learned in each memory cluster are important and interesting, but somewhat under-developed. In Figure 5 the authors show affinity maps for several memory clusters and images, and the results are somewhat convincing but also difficult to parse. I’m not sure how best to improve this analysis, but perhaps showing the same set of images for all the clusters would highlight the differences in cluster selectivity. \n\nUnfortunately the second memory cluster analysis (Figure 6) is much more difficult to understand. Here it seems like the authors are interpreting the tiny generated patches in the left-hand-side of each image in an attempt to understand what the semantic selectivity of each cluster is. While it seems clear that these patches come from different parts of the image (e.g. the left-most cluster is definitely selecting for sky in each image), the generated patches are very tiny and not very distinctive. I was left unsure about the claims made based on this Figure. I would recommend that the authors re-work this to be more clear about what they are showing, but I am afraid that I can’t really offer any specific and substantive recommendations.\n", "summary_of_the_paper": "In this paper, the authors propose a new “grandmother cell”-like memory mechanism for improving image generation performance in GANs. In short, this method clusters and stores activation vectors observed during training. Then at image generation time, activation vectors are augmented with the sum of the stored memories at the closest cluster. This seems to improve GAN performance for few-shot image generation tasks. The authors also visualize the learned memory clusters, which seems to reveal some semantic clustering.\n", "summary_of_the_review": "This paper proposes an interesting memory mechanism, and shows that it seems to help in some circumstances. However, questions about the quantitative evaluation, qualitative evaluation, and interpretation analyses leave me unsure about how much improvement the method really offers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "62TRCtulSAp", "reviewer": "Reviewer_4n7p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea of using prototye memory concept is interesting, although it is not a new idea. For example, the prototypical network for few shot learning [1]. The authors showed that the proposed MoCA module can be used to enhance the geterator in several GAN based few-shot image generation models.\n\nWeakness:\nThe authors should make the submission more clearer. The main concerns are listed below:\n1) It is still an open question that how the \"grandmother cell\"  (if exists) codes information in the brain, e.g., population coding, sparse coding, or in other forms. The sparse activity of V1 neurons in the monkey brain doesn't mean they are grandmother cells. Activities of neurons in deeper layers (L3-5) may not be sparse. The authors should be more careful about this. \n2) the memory bank idea has been proposed in many other works (the implementation detailed may be different), the authors should compared the difference of their work and others. \n3) What do the prototype semantic cells and the prototype component cells correspond to in the neocortex? Is there a correspondence, or it is just for network design? Why not only use  $K_i$ to represent the prototype memory?\n4) When transform A to a low-dimensional space, why do you use three functions $\\theta(\\cdot)$, $\\phi(\\cdot)$ and $\\psi(\\cdot)$? I only find $\\phi(\\cdot)$ in Fig.2. Also, in the main text, sometimes the authors use $\\Phi$ and $\\Psi$. All the terms should be defined clearly at the begining. \n\n[1] Snell, J., Swersky, K., & Zemel, R. S. (2017). Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175.", "summary_of_the_paper": "The authors proposed a Memory Concept Attention mechanism to improve few-shot image generation quality. They showed that the prototype memory bank improves image synthesis quality, learns interpretable visual concept clusters and improves the robustness. ", "summary_of_the_review": "The authors carried out adequate experiments to show the proposed module (MoCA) is can be used to improve few-shot image generation quality and they linked the module to the grandmother cell in the brain. However, the memory bank idea is not new and the novelty is limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper uses prototype memories for learning generative models. Inspired by the finding that there is sparse activity and complex selectivity in the supragranular layers of every cortical region, even primary visual cortex, the authors propose to use prototype memories at each level of the hierarchy, which marks their work as novel. They show superior performance in few shot image generation tasks.

The reviewers' scores were borderline (5,5,8), making this a case that required some AC consideration. The reviewers generally agreed that the paper was relevant and interesting, though the two more negative reviewers had some concerns about (1) the tests used, (2) the interpretation relative to neuroscience data, and (3) the novelty. After reading through the paper, the reviews, and the rebuttal's, the AC felt that the authors had made a decent attempt at addressing items (1) and (2), and item (3) was ultimately a subjective question. The authors were reasonably clear about what marks their work as novel, and it is certainly not *exactly* the same as previous work. Altogether, given these considerations, the AC felt that this paper deserved to be accepted, given the reasonable attempts from the authors to respond to the reviewers' concerns and an average score above acceptance threshold (though the scores did not change post-rebuttal, it should be noted).
Decision: Accept (Poster)