Few-shot Generation via Recalling  Brain-Inspired Episodic-Semantic Memory
Authors: Zhibin Duan
Lv Zhiyi
Chaojie Wang
Bo Chen
Bo An
Mingyuan Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Model
Memory-augmented Generative Model

Abstract: Aimed at adapting a generative model to a novel generation task with only a few given data samples, the capability of few-shot generation is crucial for many real-world applications with limited data, \emph{e.g.}, artistic domains.
Instead of training from scratch, recent works tend to leverage the prior knowledge stored in previous datasets, which is quite similar to the memory mechanism of human intelligence, but few of these works directly imitate the memory-recall mechanism that humans make good use of in accomplishing creative tasks,  \emph{e.g.}, painting and writing.
Inspired by the memory mechanism of human brain, in this work, we carefully design a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to assist existing generative models efficiently recall these memories during sample generation.
Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories, which can also model the uncertainty during conversion.
Then, we combine the developed VSM with various generative models under the Bayesian framework, and evaluate these memory-augmented generative models with few-shot generation tasks, demonstrating the effectiveness of our methods.

Few-shot Generation via Recalling Brain-Inspired Episodic-Semantic Memory
Zhibin Duan; Zhiyi Lv;" Chaojie Wang Bo Chen,
National Laboratory of Radar Signal Processing Xidian University. Xm' Shaanxi 710071, China xd_zhibin@163 com, xd_ ed_silly@163.o com, bchen@mail xidiian.... aedian.edu.....
Bo An Nanyang Technological University
Mingyuan Zhou
The University of Texas at Austin
Abstract
Aimed at adapting generative model novel generation task with only few given data samples, the capability of few-shot generation is crucial for many real- world applications with limited data, e.g., artistic domains. Instead of training from scratch, recent works tend to leverage the prior knowledge stored in previous datasets, which is quite similar to the memory mechanism human intelligence, but few of these works directly imitate the memory-recall mechanism that hu- mans make good use of in accomplishing creative tasks, e.g., painting and writing. Inspired by the memory mechanism of human brain, in this work, carefully design a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories assist existing generative models efficiently recall these memories during sample generation. Meanwhile, we in- troduce bionic memory updating strategy for the conversion between episodic and semantic memories, which can also model the uncertainty during conversion. Then, we combine the developed VSM with various generative models under the Bayesian framework, and evaluate these memory-augmente generative models with few-shot generation tasks, demonstrating the effectiveness of our methods.
Introduction
remarkable capability of human intelligence is its ability to quickly grasp the concepts of new objects that it has never encountered before [I]. Essentially, this rapid adaptation is achieved by utilizing the past memories stored in the human brain, which can greatly improve the efficiency of later learning 2 3 Unlike human brain, existing machine learning methods often require be trained on large amount of data when facing new tasks, motivating the research interests on few-shot learning that aim efficiently solving these unseen tasks with only few given data samples (46].. While there have been promising processes for few-shot adaptation on classification tasks, less work has been done on few-shot generation 77-10, which mainly due the challenging nature of learning the generative process with only a few samples in an unsupervised manner [1].
For existing deep neural networks, to improve their abilities of few-shot adaptation, effective methods to introduce an augmented memory module, which can store the memory of past experiences to adapt the model new task quickly 6 [3/16. However, the most widely used memory mechanism in exisiting works short-term 17 with limited capacity, which focuses on the
of the most
"qqual Contribution Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

collection of personal experience related to particular place or time. To remove the limitation of short-term memory, Zhen et al. [18] introduce the concept of semantic memory, which allows the storage of general conceptual information into meta-learning to acquire long-term knowledge for few-shot learning. Although these memory modules have separately achieved attractive progress in few-shot learning 2 19 20), there is little work to explore memory mechanism to organically combine the semantic and episodic memories, and further apply on few-shot generation tasks.
Inspired by the memory mechanism of the human brain, where both episodic and semantic memory work together to help humans understand the world around us and also accomplish creative tasks, in this work, we first develop a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to improve the few-shot generation capability of the generative models. Then, we design two important processes in VSM: 1) structured memory recall, which retrieves the relevant information of previous tasks from VSM to be applied on current task; 2) structured memory update, which continuously collects data information from training tasks and gradually consolidates the structured memories stored in VSM. Finally, we combine VSM with a branch of few-shot latent variable models, specifically neural statistician (NS) 77.. and develop series of memorr-augmeettdd latent variable models for few-shot generation.
The main contributions of this paper can be summarized as follows:
Inspired by the memory mechanisms of the human brain, we develope novel variational structured memory module (VSM) that can concurrently store both episodic and semantic memories, effectively aiding generative models in memory recall during sample generation.
Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories stored in VSM, where episodic memory can provide detailed episodes that will be converted later and stored in semantic memory.
Through incorporating VSM with NS, we develop novel hierarchical memory-augment neural statistician (HMNS), whose generative process can be specified with VAE or diffusion model, leading to VSM-VAE and VSM-Diffusion, respectively.
Extensive experiments demonstrate that our developed VSM-VAE and VSM-Diffusion can achieve promising generative performance on few-shot generation tasks.
2
Related work
Few-Shot Latent Variable Models: For few-shot generation, a series of latent variable models based on NS (7) have been developed [912], which can be further improved by adopting non-parametric formulation for the context [2I] introducing a hierarchical learnable aggregation for the input set [ increasing expressivity for the conditional prior [10, or exploring supervision 22 Specifically,
NS 77 introduces a context constant variable to specify the summary statistics of a dataset, which provides shared prior for these samples within the same dataset, contributing to handling new datasets at test time; GMN [9] defines variational model to learn a per-sample context-aware latent variable. Distinct from previous works, our method introduces a memory module to collect historical datasest information to inference the context of a new dataset.
Memory- -Aumory- Neural Networks: It has been shown that augmenting existing neural networks with an external memory module can improve their capability of few-shot learning
[23]. Specifically, neural networks mainly collect the information stored the support set of
these the current task 66] 16). which will be wiped form episode to episode [17] instead of maintaining
long-term information that has been shown to be crucial for effectively learning new unseen tasks. For instance, variational semantic memory [88 is proposed to accumulate and store the semantic information from previous tasks for the inference of the prototypes of new tasks. Distinct from previous methods only constructing semantic memory for the prototype network, in this paper, we develop structured memory module to store the historical information at both the episodic and semantic sides, which can largely enhance the model's capability of few-shot generation.

---

1-1.dh
L...
j.L..
1-1...N
(a) NS
(b) HNS
(c) VSM-VAE
(d) VSM-Diffusion
Figure 1: The generative process of (a) neural statistician (NS), (b) hierarchical neural statistician (HNS), (c)
variational autoencoder (VSM- VAE), and (d) nemory-augment difussion model (VSM-Diffusion), where æi.j denotes the observed data sample, Zig is the latent variable at the /-th hidden layer, and c, and m are the context variable and memory variable of dataset D,, respectively. More details can be found in Section 3.2
Memory-Augmented Neural Statistician
3.1 Preliminary
Few-shot Generation:
Few-shot generation (2 borrows a similar setting of few-shot learning 4
for a more challenging task to generate objects given only a few samples that object test time.
Specifically, given a set of training datasets Datd = {D:) where each dataset D; = { f==1 consists of J; data samples, the goal of few-shot generation is to train a generative model (æ) that trained on given datasets Dota but can be quickly adapted new unseen task given only few data samples collected from a new dataset Dnew-
3.2 Memory-Augmenter Neural Statistician
Through incorporating memory mechanism with few-shot latent variable models, specifically NS 7 in Figure I(a), in this paper, we develop a novel memory- augmented NS (MNS) for few-shot generation.
The design of MNS aims at introducing a memory module to store and update the data information at various semantic levels, which can be recalled when facing new generative task. As shown in Figure I[c) the generative process of hierarchical MNS is formulated
Generative Process of MNS:
as:
(Dtd)) ///
J pa (Day Rigs Ci m.)
==1Ø=1
N J:
N
N
IIII (= ==
(c)) II (a(c: m.) II (im,,dzz, de, dm4. ==1
==1
To clarify the meanings of zt C, and m,: 2., denotes the per-sample latent variables of the jth sample in dataset D,, c, denotes the task-specific context variable of dataset D,. whose detailed definition can be found in (7]. and m, denotes the task-specific memory variable, which stores the data information of previous tasks that will be adapted to the new task.
3.3 Variants of Hierarchical Memory-Augmented Neural Statistician
Through specifying the generative process of HMNS and replacing its memory module with the variational structured memory (VSM) as described in Section 4 we can obtain series of model

---

variants, such as VSM-HVAE and VSM-Diffusion, both of which can achieve promising generative performance on few-shot generation task.
3.3.1 VSM-VAE
Generative Process of VSM-VAE: To capture the hierarchical data information at various semantic levels, we extend the latent variables in Eq. (I to hierarchical version, specifically ži.j { 1
Ciji ( and mij {m و which will be included in a variational autoencoder (VAE) with L hidden layers, leading to VSM-VAE shown in Figure [[c] Under the generative framework of MNS, using the same symbol notation in Eq. (T. the per-sample generative process of æijj in the
dataset D, can be formulated as
po(x..)) = J po par. 2i.j (2 cc:)p((c; m.)p(m:) dzı,, de; dm
(2)
where (z |c,), pe(c, m4), and p(m,) are formulated as the product of corresponding hierarchi- cal latent variables as follows
L
Po(zi.) (a) = II
po(z))
(1+1) c()),, po(c; m.) = II
po(c le (+1)) .m!"),
(3)
1=1
1=1
L-1
po(m:)
II po(m
(
|m
(1 (((()))) p(m! ) N(,
Training Objective of VSM-VAE: For VSM-VAE, the training objective is to maximize the evidence lower bound (ELBO) of a set of training datasets (D:) which can be expressed as
EQ [oggo c()) - Eq log go(c, Di, m.)
Io Di, M) (o(m.)
CELBO
Σ
- BQ log
(5)
po(cs |m,)
The first item in Eq.
5 can be further written
log 9 g !1) Di,s 2in cc
(1+1)
Eq [logpe(DD Pe Dj |-]
EQ log (ogpo(assl1))))
Σ BQ
po(zi.) |co 2ig
(1)
((+1)
1=1
(0(c,1-)
11) Do (1+1) m. (1) EQ log
(o(C)"
Eq
log po(cs] m,:)
Σ
(6)
po(c!") ( (+1)) m (1)
1=1
Yo (m, (m):))
(m (1) DDi M)
Eq log
EQ
log
po (m m, (++)))
1=1
and Q denotes the variational distribution defined by the inference network of VSM-VAE
Q -=IL=_ 90(2
(1+1) m.j. 2,5
w ("))((e() Dr, (.) (m) DD, M),
(7)
where M indicates the memory module to store the semantic information collected from previous tasks, and will be introduced in Section
3.3.2 VSM-Diffusion
Generative Model of VSM-Diffusion: Further, through specifying the generative process of HMNS with the diffusion model, can obtain the VSM-Diffusion, shown in Figure I(d) Distinet from the hierarchy of latent variables in VSM-VAF, we share the same context and memory variables across
all hidden layers of a diffusion model, specifically c; = c and m, m. for I (1, L}. Thus, under the generative formulate MNS, using the same symbol notation in Eq. (I.. the par-sa generative process of æ,,j in the dataset D, can be formulated as
1) po(x).))= = / po(...)) 1 HE ca)po (m:::::mmppmm.. d(x L de, dmi,

---

where the backward diffusion process is consistent with that of conditional diffusion model 224 25 formulated po (aig)iz ci) po( 1 ci)
L-1
II
po (1 (1(1)) cs) cs)po((r))))
(8)
Training objective of VSM-Diffusion: For VSM-Diffusion, the training objective is derived from maximizing the evidence lower bound (ELBO) of a set of training datasets {Di)==> which is consistent with Eq. (5). Similar to regular DDPMs, the first term can be decomposed into a summation of L terms, each of which can be computed independently, and can be estimated as
J1
Po ( f (1)
J
I. E + Li
Eq [log pa [D Ci2 m,)]
Σ
BQ
log
L30
go (1 | x
1=1
j=1
1=2
where LL is a fixed constant, Lo the expectation of likelihood term and the intermediate term L can be formulated as
2al2! (1-1)
1 (=
log
(1-1) po(o).) 2 L Ci)
(1)
4 Variational Structured Memory
equipped with the mechanism of memory recall and update in Sections 4.2 and 4.3 respectively. The basic intuition of VSM is inspired by the memory mechanism in the human brain: 1) there are two advanced forms of memory in the human brain: semantic memory allows the storage of general conceptual information and episodic memory allows the collection of detailed episodes; 2) episodic memory can provide detailed episodes that will be converted into conceptual information and stored in semantic memory. The structured memories stored in VSM will be recalled to infer the posterior of task-specific memory, formulated as:
In this section, we will introduce the storage architecture of VSM in Section 4.1
(m(m;
(10)
which will be used in the inference process of either VSM-VAL VSM-Diffusion.
4.1 Structured Memory Storage
As shown in Figure 7 there are N blocks (categories) in the VSM module, denoted as M [M,)-= and each memory block M,, consists of both semantic memory M((") and episodic memory M
Semantic Memory Storage: The semantic memory in the human brain can provide conceptual information for quickly learning concepts of new object categories by seeing only a few data samples. Following Zhen et al. [18], for each block (category) of semantic memory in VSM, we try to store the conceptual information by averaging the latent representations of data samples belonging this category, contributing to lightweight storage and fast lookup. To model the uncertainty, we model the nith block (category) of semantic memory with Gaussian distribution, formulated
( (M J M( diagg,
(11)
where ,, nn Rdd denotes the d-dimensional mean and variance vectors, respectively.
Episodic Memory Storage: Distinct from semantic memory allowing the storage of general con- ceptual information, episodic memory tends to focus on storing more detailed and vivid experiences or episodes. The design of episodic memory is inspired by previous works [26)30), and the main idea to employ query memory module collect subset of representative data samples for each block (category) in VSM. Specifically, the n-th block (category) of episodic memory is set of T embedding vectors of data samples:
( (M >
M( =
(12)
where is the storage limit of each block (category) of episodic memory, and the dimension of ( Rd is the same the latent representations of data samples.
5

---

Set Festure
KNN Attention
neighbor lookup
h RR
(Query)
(Key, Value)
Semantic Memory JR
Semanti
M
M;
My
x
Memory c
Ephodie Saaoory
Eptoddic
M R
Figure 2: The workflow of memory recall mechanism in variational structured memory (VSM) module M, which stores both semantic memory ( and episodic memory M(e),
4.2 Structured Memory Recall
Given a query dataset D; = { 1 which can be treated as a new dataset Dnew for previous tasks, the mechanism of memory recall aims effectively using the memory module quickly adapt a model trained on previous tasks to new unseen task. The workflow of structured memory recall in VSM has been depicted in Figure which includes both the recalls of semantic memory ( and episodic memory M(e), resulting in two kinds of task-specific memories, denoted m; and m; respectively.
Semantic Memory Recall: For new few-shot generation task on the dataset D; (Dnew), the memory recall in VSM will first select the top-K relevant blocks (categories) of semantic memory
as the candidates, denoted as (M jkmi ) and then use an attention mechanism to aggregate these
candidates into a task-specific semantic memory m; (e) calculated as
where the attention weight of n-th can be
( h.) exp (u,, /
(13)
where g(-) can be treated as similarity function, and h, indicates the semantic representation of dataset D; and can be obtained by averaging the latent representations of data samples in D,.
Then, with the top-K relevant blocks of semantic memory {M, (+) K jk=i in hand, we can further aggregate these candidates with their attention weights, formulated as:
(+) h: ΣK 9 (M). hi)
(M{
K mi Σ
M(()) ~ diag(())))))
ALa
(14)
k=1
where m(e) E IRd has gathered the relevant information from the semantic memory M( for the generation task dataset D,.
Episodic Memory Recall:
After selecting the top-K relevant blocks of semantic memory (M{)) ) 1k=
we can further dive into recalling episodic memory, which is more specific and detailed. In practice, each block of episodic memory contains T embedding vectors of data samples M (M, (e) 1t=
and we can finally obtain K * embedding vectors of data samples. After that, following a similar process recalling episodic memory, we can further aggregate these obtained embedding vectors with their attention weights as
M
(c) M.thhhi
12
m Σ
T Σ
M E,t3
A
9
(15)
1=1
ΣK ΣT= (M(e) h:
where g(-) and h; have been defined in Eq.
13). and the obtained m (e) IRd has gathered the relevant information from the episodic memory (e) for the generation task dataset D,.

---

4.3 Structured Memory Update and Consolidation
Inspired by the characteristics of human memory 31 we develop a novel semantic memory update mechanism that considers the data information from previous semantic memory M(+), episodic memory M(e), and new task D;- Specifically, the semantic memory module is empty at the beginning of learning, and each unseen block (category) M.." will be initialized with the mean and variance of the latent representations of data samples, which is collected from the current task Dj and belongs n-th category. For those seen categories, their semantic memory can be updated with the data sample from the current task D; using self-attention mechanism.
Semantic Memory Update:
Following Zhen et al. [8 we construct a graph with respect to the memory M( for updating, which consists latent representations of Jn, data samples that belongs to n-th category and T data samples
stored in episodic memory M denoted as Hl, {mn,1, 2 .. Jn.J., n,1 ,M n,7h. The
M
mean of the semantic memory M! can be updated as:
g (Mi") Hms)
nese M
axpold M
a) MM PM
JutT Ejnt Bag.inns Bn. Ex-T y (M." Hon))
(16)
where a E (0, 1) is hyperparameter; Pn, indicates the attention weight between j-th node features and the source node M g(.) is the similarity function defined in Eq. (33 This operation allows the useful information to be retained in memory while erasing less relevant or trivial information.
After that, we employ the incremental update strategy of the Gaussian distribution variance of semantic memory M( as follows:
34
to update the
new f a M
2 old "
old (M!) )j+ +1 (1-1aa) -0) (00.. + (M Hn)
(17)
where H, and H. are the mean and variance of the set of node features Hn, respectively.
Episodic Memory Update: Inspired by the fact that it is difficult for person to forget those things he/she often recalls (and vice versa), design frequency-basee episodic memory update stragy. Typically, for each data sample ( stored in episodic memory M we use a frequency matrix, denoted as Mi to record how many times ((e)) n.z was recalled during training. Then, for updating
M (M 1 if the current episodic memory block has not been filled yet, we will directly append the latent representation of new data sample the M when a new task arrives; Otherwise, we will update M by replacing the least used data sample in the block if its capability exceeds the limit T. We note that the frequency matrix CM(Ce) will be reset the zero matrix after each update of episodic memory. The detailed memory updating process algorithm can be found in Appendix A
5
Experiments
5.1 Experimental setup
Datasets:
The experiments are conducted on five widely used benchmark datasets of various
sizes. including binarized Omniglot (2). MNIST 35), DOUBLE-MNIST [36), CelebA (37 and FS-CIFAR 100 [38] The split of training/testing set (also known background-evaluation split) follows Lake et al. (2). in which scenario all the samples at test time are from new classes. We resize all the images of binary datasets to 28x28, FS-Cifar 100 to 32x32, and CelebA to 64x64.
Baselines: 1) For VAE-based models, we utilize NS-related models, specifically focusing on two NS variants: Convolutional Neural Statistician (CNS), where the latent space is shaped with convolu- -ierarchica-- Aggregation-VVariatinaaal iitiatiionllAuttennoodd (SCHA- VAE) 224, which employs additional hierarchy over the context latent variable The CNS and SCHA- VAE can naturally extended under the framework of VSM-VAE, leading to VSM-CNS and VSM-SCHA, respectively. Meanwhile, we create several variants by fusing either semantic memory or episodic memory with CNS or SCHA-VAF, as appropriate, further demonstrate the efficacy of the proposed variational structured memory; 2) For Diffusion-bars based models, consider both
tions at given resolution;
7

---

Table 1: Few-shot generative evaluation of VAE-based models various datasets with the set size 5, including Omniglot, Double-Mnist, FS-Cifar 100, and Mnist (trained on Omniglot and tested on Mnist.). For all datasets, memory size to be the same as the number of categories appearing in training data.
the episodic memory size of all methods to 5 and the semantic
Ommiglat-ns
Double-mnistl
PS-Cifar100
NELBO
NLL
NLL
NELBO(bpd) NLL/bpd)
69.2
105.1
NS CNS
96.6
92.4
67.3
60.2
118.7
101.2
92.9
89.7
66.5
54.3
114.1
90.4
9.979
9.803
CNS + Semantic
84.4
72.4
56.7
50.2
107.3
88.5
9.952
9.745
CNS + Rpisodic
83.9
71.8
54.3
51.1
97.8
80.1
9. 9.946
9.741
VSM-CNS(our)
81.1
68.2
47.8
40.6
92.9
77.0
9.872
9.711
SCHA-VAR
114.7
SCHA
77.7
10.317
10.215
SCHA
10,292
VSM-SCHAA(ur)
74.0
56.8
42.4
36.5
89.9
68.2
10.219
10.041
Table 2: Few-shot generative evaluation of diffusion- models various datasets with the set
size 5, including Fs-Cifar 100 and Celeba. (FID: Frechet score; sFID: spatial FID; P: precision; R:
recall.)
Celcha
4
P1
FID J
DDPM
62.84
28.91
0.58
0.40
14.02
27.46
0.71
0.39
sDDPM
45.50
29.87
0.54
0.46
12.87
26.44
0.72
0.37
vDDPM
62.58
27.50
0.58
0.41
13.01
26.24
0.73
0.39
<FSDM
63.73
28.85
0.58
0.38
13.52
27.75
0.71
0.38
VSM-vFSDM(um
44.11
23.11
0.59
0.46
12.27
25.79
0.72
0.40
FSDM
35.07
20.95
0.62
0.53
10.11
24.58
0.73
0.39
VSM-FSDM(ou))
33.21
18.32
0.68
9.25
23.21
0.77
0.43
unconditional and conditional diffusion models as baselines, where we use the DDPM 39] as the unconditional baseline and other three conditional diffusion models: VDDPM, where the context is latent variable and the dataset diffusion model where employees a different way to extract and aggregate set-information using ViT [40] stack all the samples on the channel dimension; sDDPM adapting ideas in Sinha al.
generated conditioned on e; VFSDM 24 another variational
41 without contrastive learning, where a ViT encoder splits the image in patches and processes them jointly.
Model Settings: The details about model settings can be found in Appendix B
5.2 Quantitative experiment
VAE-based models: To evaluate the generative properties of VAE-based models, we use the evidence lower bound (ELBO) to approximate the log marginal likelihood with 1000 importance samples (NLL).. Our qualitative experiments are performed on Omniglot, BOUBLE-MNE
MNIST, and FS-Cifar The outcomes of the experiment are displayed in Table 5.2 It is clear that VSM- CNS/VSM-SCHA can achieve significant improvements over its baseline models, proving that VSM could be a promising mechanism for few-shot generation. Additionally, by incorporating either semantic episodic memory into CNS/SCHA- VAE, the resulting models can also achieve better performance, proving both memory types can improve the model performance of few-shot generation.
Diffusion-based models: As exhibited in Table we employ the metrics below to assess the generative capacity diffusion-based models, including: FID [42 for sample quality, sFID (43 to capture spatial relationships, and Precision and Recall 44 for measuring variety and mode coverage. From the results, VSM-Diffusion beats the other unconditional and conditional baselines on both two datasets, demonstrating the effectiveness of VSM few-shot generation.

---

(a) Query Set
(b) Generative Samples
(c) Query Set
(d) Generative Samples
Figure 3: Visualization of few-shot generative samples using VSM-Diffusion with the set size 5. The query set and generative samples on the left side are from FS-Cifar 100, while those on the right side are from Celeba.
Query Set
(b) Recalled Semantic Memory
(c) Recalled Episadic Memory
Figure 4: (a) Query Set: (b) Recalled blocks (categories) of semantic memory. each of which is represented by the images of corresponding category; (c) Recalled samples of episodic memory.
85
85
NLL
NLL
75
NELDO
MEInO.
65-
50
3
19
10
10
Eptoddic memory site
Eptoddic memory size
Episodic momory size.
(a) Omniglot
(b) Doublemnist
(c) Mnist
Figure 5: Few-shot generative evaluation of VAE-based models on various datasets with different episodic memory sizes.
5.3 Qualitative experiment
Few-shot image generation:
The generation results of VSM-Diffusion are exhibited in Figure 3 From the results, can find that memory-augmenter models generate class-specift samples through successfully extracting content information and realistic classes that are both complicated and varied. Limited by pages, additional fhw- generative experimental results Omniglot, CelebA, and FS-Cifar 100 can be found in Appendix C.2
Table 3: Ablation study on the update strategy variational structured memory (VSM).
Omniglot ns
Duuble Maist
Mnist
NELBO
NLL
NELBO
NLL
NELBO
NLL
W/o Episodic memory
75.4
58.2
45.3
40.1
91.1
69.5
Frequency-bbas Combined strategy
74.8
58.2
37.7
91.7
56.7
36.5
Memory visualization:
To further verify the effectiveness of the proposed VSM, we conduct experiments of memory visualization here. For instance, given a new set, as shown in Figure [(a) the images of recalled semantic memory corresponding to the class are shown in Figure (bb) and the images of recalled episode memory are shown in (igure((c) The visualization results demonstrate that VSM can recall related image classes and samples to the query set in an effective way.

---

5.4 Ablation study
Effect of memory size: To investigate the affect of memory size in episodic memory, we use the VSM-SCHA model typical example. We train model on Omniglot data, and evaluate it with the Omniglot, DOUBLE- --NIST, and MNIST datasets for unknown classes. The size of the episodic memory is raised from 0 to 10. The results in Figure 5 show that the model performance improves with the increase of episodic memory size, further demonstrating the importance of episodic memory in few-shot generation tasks. We note that the developed variational structured memory will reduce to variational semantic memory [18] when the episodic memory size is adjusted to
show
Effect of memory update process: We conduct two variants of VSM-SCHA to investigate the benefits the proposed structured memory update strategy. For the first variant, its semantic memory update strategy does not consider episodic memory, denoted as W/o Episodic memory; For the second variant, episodic memory update strategy does not employ frequency- strategy but uses first-in, first-out queue-based memory update method, denoted as W/o Frequency-based The experiment results in Table 3 can verify the effectiveness of the proposed memory update strategy.
6 Conclusion
In this study, we develop a novel variational structured memory module that can concurrently retain generic semantic structure about a category and specific information about sample, the former being achieved by semantic memory and the latter being caught by episodic memory. Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories stored in VSM, where episodic memory can provide context information that will later be transferred and stored in semantic memory. Through incorporating VSM with few-shot latent variable models, we develop novel hierarchical generative process can be specified with a VAE or diffusion model, leading to VSM-VAL and VSM- Diffusion. Extensive experiments demonstrate that our developed VSM- VAE and VSM-Diffusion can achieve promising performance on few-shot generation tasks.
neural statistician (HMNS), whose
Acknowledgments
This project was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22160 160; in part by the 111 Project under Grant B B18039; in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSJ23016; Additionally, this project was supported in part by the National Research Foundation, Singapore under its Industry Alignment Fund Pre- positioninn (IAF-PP) Funding Initiative.
References
[1] Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
[2] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(62666 2015.
[3] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:/801.08933 2018.
[4] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[5] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. International conference on machine learning, pages 1126-1 1335. PMLR, 2017.
10

---

[6] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[7] Harrison Edwards and Amos Storkey. Towards arXiv:1606.02/855 2016.
neural statistician. arXiv preprint
[8] Danilo Rezende, Ivo Danihelka, Karol Gregor, Daan Wierstra, et al. One-shot generalization in deep generative models. In International conference on machine learning, pages 1521-1529 PMLR, 2016.
[9] Sergey Bartunov and Dmitry Vetrov. Few-shot generative modelling with generative matching networks. International Conference on Artificial Intelligence and Statistics, pages 670-678. PMLR, 2018.
[10] Luke B Hewitt, Maxwell I Nye, Andreea Gane, Tommi Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv: 1807.08999 2018.
[11] Giorgio Giannone and Ole Winther. Scha-vae: Hierarchical context aggregation for few-shot generation. In International Conference on Machine Learning, pages 7550-7569. PMLR, 2022.
[12] Zhibin Duan, Yishi Xu, Jianqiao Sun, Bo Chen, Wenchao Chen, Chaojie Wang. and Mingyuan Zhou. Bayesian deep embedding topic meta-leamer. In International Conference on Machine Learning, pages 5659-56700 PMLR, 2022.
[13] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with megeee-a neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[14] Eukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprini arXiv: M:X3:3/703.03222 2017.
2017.
[15] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International conference on machine learning, pages 2554-2563. PMLR, 2017.
[16] Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International conference machine learning, pages 3664-3673. PMLR, 2018.
[17] Augusto Buchweitz. Models of working memory: Mechanisms of active maintenance and executive control (1999). Ilha do Desterro A Journal of English Language, Literatures in English and Cultural Studies, (43):193-206) 2002.
[18] Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees Snoek, and Ling Shao. Learning to learn variational semantic memory. Advances in Neural Information Processing Systems, 33: 9122-9134, 2020.
[19] Brenden Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[20] Tomer D Ullman and Joshua B Tenenbaum. Bayesian models of conceptual development: Learning as building models of the world. 2020.
[21] Mike Wu, Kristy Choi, Noah Goodman, and Stefano Ermon. Meta-amortized variational inference and learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6404-6412, 2020.
[22] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprini
2018
[23] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[24] Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models, arXiv preprint 1544, 2022.

---

[25] Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Bjöm Ommer. Retrieval- augmented diffusion models. Advances in Neural Information Processing Systems, 35:15309- 15324, 2022.
[26] Arslan Chaudhry, Marc" Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint 1/Xiv:8//2.004200 2018.
[27] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[28] Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomênech Badia, Gavin Buttimore, Charles Deck, Joel Z. Leibo, and Charles Blundell. Generalization of re- inforcement learners with working and episodic memory. Advances in neural information processing systems, 32, 2019.
[29] Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning. Advances in Neural Information Processing Systems, 33: 1023-1035, 2020.
[30] Jun He, Richang Hong, Xueliang Liu, Mingliang Xu, Zheng-Jun Zha, and Meng Wang. Memory- augmented relation network for few-shot learning. In Proceedings of the 28th ACM International Conference Multimedia, pages 1236-1244, 2020.
[31] Endel Tulving. Episodic and semantic memory. 1972.
[32] Larry R Squire and Stuart M Zola. Episodic memory, semantic memory, and amnesia. Hip- pocampus, 8(3):205-21,, 1998.
[33] Daniel L Greenberg and Mieke Verfaellie. Interdependence of episodic and semantic memory: Evidence from neuropsychology. Journal of the International Neuropsychological society, 16 (5):748 15,748-733, 2010.
[34] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[35] Yann LeCun, Lawrence D Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs A Muller, Eduard Sackinger, Patrice Simard, et al. Learning algorithms for classification: A comparison on handwritten digit recognition. Neural networks: the statistical mechanics perspective, (61(27 1995.
[36] Shao-Hua Sun. Multi-digit mnist for few-shot learning, 2019. URL httph::/tttthhhiee com/ Banohua0116/Multiiiiigo.
[37] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 7730-3738, 2015.
[38] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
[39] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
[40] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
[41] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances Neural Information Processing Systems, 34:12533-12254 2021.
12

---

[42] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[43] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.0384) 2021.
[44] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
13

---

A
Memory Recall and Update Algorithm
The procedure of memory recall and update algorithm is summarized in Algorithm, 1
Algorithm 1 Semantic Memory recall and update algorithm
Input:
Given task D;; Semantic memory M and episodic memory M(e);
Recall task-specific semantic memory according to Eq. 14 and episodic memory according to Eq. 15:
The semantic memory for the given task is null: then
Computing the mean and variance for the semantic memory with given task D;;
else
Update the semantic memory according to Eq. and Eq. 17
end
if The episodic memory for the given task null: then
Append episodic memory with given task Dj; else
Update the episodic memory according to frequency matrix C; end if
Infer variational posteriors for the latent variables of different layers f H
Model Design: In most memoryybaseed models, the images D; = {x., 1 are fed into an encoder
to produce image features {hi 15- E Px1, which are then represented as vectors. These vector representations are utilized for both recall and update operations in the model. Nonetheless, due to the high dimensionality of images, relying solely vectors lead loss of valuable information.Be drawing inspiration from the approach proposed in the Vision Transformer, we can overcome these challenges and obtain more accurate memory information. We begin by obtaining image features, denoted by { RP Px then calculate the average of the set dimension resulting in
J. Σj=1 haj Afterwards, we reshape the size of the image features to { E E N where D denotes the patch size. When updating M((*) RRDD, we employ the resized feature h R(4xN))xD for graph attention. As for Mf() E BLLNND, where L denotes the total number of episodic memory, we directly replace the least recently used patch in Mf with ..r.BBy using per-patch recall and update, we obtain information from different semantic levels. This makes the Semantic Memory more suitable for task generalization, while the Episodic Memory is more capable of capturing sample-specific details.
VSM-SCHA In SCHA-VAE and VSM-SCHA, we shared routing path between and Dil (1+1) ((+1) (1) where 21+1 and C++ are samples from layer I. We additionally
(o D;, c -2, remove any form of normalization (batch/layer norm) in the hierarchical formulation
Memory size Across datasets, we set the size of each semantic memory block (category) in VSM-based model to 1, and each episodic memory block to 5. For models that exclusively depend on episodic memory (CNS + Episodic, SCHA Episodic, and Episodic-diffusion) we establish memory size of 10000, with the episodic memory functioning first-in, first-out queue.
B Details of Model Design
C Additional Experiment Results
C.1 Quantitative Comparison of Generative Samples
In this section, we apply VSM to FSDM, a conditional diffusion model where e not a variational variable. To align with FSDM, only use the following update rule for the semantic memory:
14

---

Input
Output
Hnael RALL
Dmeaee Ber.
Inoor Bol.
Semento Co
Loodo mewert
Baruul Aud
Mere BALUL
Snae Bink]
Exd
212
202
2rd
213
bargu
301
Figure 6: The process of memory recall mechanism in VSM-SCHA.
Inport
menertimmecyy
Teclack
meeeey
30353 Trirsfome:
InfronceSlok.cc
Resbock
Tedaoc
3p398
V
Decoder Block
V/ Oulput
Figure 7: The process of memory recall mechanism in VSM-FSDM.
9 (M/,") Hns)
Ej1t JutT 1=1 Bag Hug: Bas
new MD
old M! +
a) Paalo Pait
(18)
Io++ 9 (M(.), Hos
Table 4 presents the results of VSM-FSDM Fs-Cifar100 and celeba datasets. It can be observed that VSM performs well even without using variational latent variables, demonstrating its strong performance and scalability.
C.2 Additional Visualization of Generative Samples
More visualization results of generative samples can he found in Figures
10 and 11
Table Few-shot generative evaluation diffusion-basse models on various datasets with the set size 5, including Fs- Fs-Cifarl00 100 and Celeba. (FID: Frechet score; sFID: spatial FID; P: precision; R: recall.)
PS-Cifar100
Celeba
FDD
sFDD
Pa
R
FID 1
sFID J
P1
R
DDPM
62.84
28.91
0.58
0.40
14.02
27.46
0.71
0.39
FSDM
35.07
20.95
0.62
0.53
10.11
24.58
0.73
0.39
VSM-PSDM
33.21
18.32
0.68
0.61
9.25
23.21
0.77
0.43
15

---

Table 5: Ablation study diffusion fiffus models
FS-Cifar100
Celeba
FID
sFID 4
P1
R
FID
sFID
Pr
R
FFDD
63.73
28.85
0.58
0.38
13.52
27.75
0.71
0.38
vFSDM semantic
60.21
27.63
0.58
0.42
12.98
26.01
0.73
0.39
vFSDM + spisadic
52.13
25.25
0.59
0.44
12.36
25.97
0.72
0.41
VSM-diffusion
44.11
23.11
0.59
0,46
12.27
25.79
0.72
0.40
(a) Query Set
(b) Generative Samples
Figure 8: Visualization of few-shot generative samples using a VSM-Diffusion with the set size on Celeba dataset. (Conditioning set and samples from in-distribution classes).
D Border Impact
The impact of our research can be highlighted in two key aspects:
For Few-shot Learning Task: Our structured memory has demonstrated high level of effectiveness in storing and managing telkteelated information, enabling efficient transfer of prior knowledge from previous tasks to new ones. By combining semantic and episodic information storage, our model enhances its ability to capture both task-level information and fine-graines image-level details. Diverging from traditional few-shot models, our approach leverages the stored and recalled memory to extract more valuable insights while minimizing the influence of irrelevant information on new tasks.
Our modularized memory offers seamless integration into wide range of generative models, such as diffusion models and VAEs, while also easily merging with latent variables. This flexibility allows for efficient utilization of the memory component within generative framework. In addition, our structured memory can be divided into multiple semantic levels, providing the ability to learn various refined semantic layers. This hierarchical organization of memory facilitates capturing complex dependencies and relationships across different levels of
For Deep Generative Models:
16

---

(a) Query
(b) Generative Samples
Figure 9: Visualization of few-shot generative samples using a VSM-Diffusion with the set size on Celeba dataset. (Conditioning set and samples from out-distribution classes)
abstraction. By incorporating multiple semantic layers, generative models can potentially achieve enhanced performance and generate more meaningful and coherent outcomes.
E Limitation
The proposed model variant, VSM-Diffusion, based on the denoise diffusion model, which requires multiple steps generate samples. The framework can be extended other diffusion models, such as implicit denoise diffusion models, to accelerate the generative process and improve its efficiency.
17

---

(a) Query Set
(b) Generative Samples
Figure 10: Visualization of few-shot generative samples using VSM- DEM--Dffusion with the set size 5 on FS-CifarlO0 100 dataset. (Conditioning set and samples from in-distribution classes).
18

---

(a) Query Set
(b) Generative Samples
Figure 11: Visualization of sho- generative samples using VSM-Diffusile with the set size on FS-CifarIO0 100 dataset. (Conditioning set and samples from out-distribution classes).
19


Reviews:
[{"review_id": "7LYQiGylaH", "reviewer": "Reviewer_zaLT", "summary": "The authors propose to incorporate 'episodic' and 'semantic' memory inspired by the human memory system into existing generative modeling frameworks to improve few-shot generation abilities. Specifically, they propose a variational structured memory module (VSM) and show its incorporation into existing VAE based models and diffusion based models can improve few-shot generation across multiple datasets. The memory mechanism is introduced as a prior to the context variables in traditional neural statistician (NS) and diffusion modeling frameworks. The ‘semantic’ memory is modeled as a ‘general’, lightweight storage with quicker retrieval (single embedding for each category) while ‘episodic’ memory is modeled to be more detailed/'vivid' (a set of embeddings for each category) and provide context relevant information. Learning involves an attention-based recall/lookup from prior memory states and subsequent update of the semantic and episodic memory for each category.", "questions": "Please see weaknesses section above.", "limitations": "A brief limitation section is provided in the appendix L449. Perhaps it could be expanded to discuss potential negative applications of generative/diffusion-based methods.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The integration and formulation of proposed memory modules (and relevant storage, recall and update mechanisms) to improve few-shot generation is relatively novel.\n2. The experiments are performed on established datasets and the results support the claim that the proposed VSM module can benefit performance on few-shot generative/diffusion-based tasks.\n3. The method and underlying components are largely adequately described in section 3 and 4 with relevant references or background provided.", "weaknesses": "1. The work's claim of modelling the memory mechanism like a 'human being' for few-shot generation tasks seems far-fetched and not adequately backed by literature or explanation. While episodic and semantic memory indeed are components of human memory (and some references for this are provided), it is not clear how they are related to static image few-shot generation (specifically, episodic memory is largely related to an agent's experiences/events and seems more relevant to episode/reinforcement learning based tasks/frameworks). \n\n2. Further, it remains unclear what is the authors mean when they mention 'context' in relation to the proposed semantic and episodic memory. E.g. on L146, it is mentioned 'semantic memory in human brain can provide context information'; while L139 mentions 'episodic memory can provide context information'. So it is unclear if both semantic and episodic are to provide 'context' and if so what does 'context' mean here? Again, in relation to my first point, backing statements such as L146 with references would be beneficial.\n\n3. Experiment setup: It is unclear whether all models were evaluated with the same vision encoder and processing. (In appendix L401 authors mention they draw inspiration from vision transformer in processing image features and suggest their process differs from existing memory-based models). Were other models evaluated in a similar setup for fair comparison? Otherwise, performance benefits might be due such differences in experimental setup and not necessarily the proposed memory method itself.  \n\n4. Further, authors should consider indicating the parameters of their method and other relevant factors to provide a better comparison to existing models (e.g. see Table1 in SCHA-VAE: Hierarchical Context Aggregation for Few-Shot Generation Giorgio Giannone, Ole Winther International Conference on Machine Learning, ICML, 2022). It is currently unclear whether the performance benefits may merely be due to more parameters or if it is due to the proposed method.\n\n5. While source code is provided in appendix, would it be released publicly for reproducibility? (currently there is no indication). Also, how many trials were performed for each experiment? \n\nMinor:\n1. The writing in introduction has minor grammar mistakes and missing words / wrong phrasing (e.g. L9: 'to assistant' -> 'to assist'; L18: 'that never encountered before' -> 'that it has never encountered before'; L24: 'which mainly' -> 'which is mainly'; L49 'to assistant' -> to assist', etc)\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions.\n\nQ1.  VSM for static image few-shot generation. \n\nA1: Thanks for your valuable question. Actually, the motivation of our work is to develop a series of memory-augmented generation models to imitate the creative process of humans. For instance, when a painter is required to paint an image with a bird, he will first recall the conceptual information (semantic memory) of a bird equipped with a series of realistic bird samples (episodic memory), and then fuses these two kinds of memory information to accomplish this creative task. And this is also the reason why we want to imitate the memory mechanism of human to redefine the generative process of existing generative models. Similar idea of introducing memory mechanism can be found in other works [1,2,3,4].\n\nMoreover, memory modules have also been widely used in few-shot learning tasks. For example,  LSTMs trained to meta-learn can quickly learn never-before-seen quadratic functions with a low number of data samples[5]. And [6] builds an effective memory augment meta-learning framework for few-shot learning with an attention module.Recently, [7] proposed variational semantic memory and applied it to the few-shot classification task, where the intuition behind this is that the model can utilize semantic information (semantic memory) gained from past tasks to solve a new task. These two mentioned aspects inspire us to develop a variational structured memory module and apply it on few-show generation task.\n\nAs for the episodes in RL, in our understanding, the episodes of RL, which will be stored in buffer, are only used to update the policy network, but won't be recalled during testing. But we believe that the idea of introducing memory module can be also extended for improving the performance of existing RL-based methods.\n\nQ2. it is unclear if both semantic and episodic are to provide 'context' ......?\n\nA2. We're sorry for the confusion caused by our unclear description. Indeed, there are two advanced forms of memory in the human brain, specifically semantic memory allows the storage of general conceptual information [7] and episodic memory allows the collection of detailed episodes [8]. We have highlighted these descriptions in the main paper and modified the sentence ``episodic memory can provide context information that will later be store in semantic memory''  `` into ``episodic memory can provide detailed episodes that will be converted into conceptual information and stored in semantic memory``. We have also modified other unclear descriptions about context information in our paper.\n\nQ3. Fair comparison? \n\nA3. Sorry for the misunderstanding. Actually, the only difference between our method and other baselines is the introduction of the proposed variational structure memory module and the remained factors, like network structures, experimental settings and evaluation metrics, \nare exactly the same as described in their papers. Specifically, we introduce the variational structure memory module into VAE-based models [9] and diffusion models [10], leading to VSM-VAE and VSM-diffusion, respectively. And we directly follow and implement our method on their released code to make a fair comparison, which can be checked in our submitted supplemental material. We have also listed the network structures in comparsion as follows:\n\n| Model |  Encoder|\n| -------- | --------- |\n|CNS |ResNet|\n|SCHA | ResNet|\n|VSM-CNS |ResNet|\n|VSM-SCHA | ResNet|\n|sDDPM | ViT|\n|vDDPM | Unet|\n|vFSDM  | ViT|\n|VSM-Diffusion | ViT|\n\nThe description in Appendix L401 is only used to illustrate the implementation details of variational structure memory module, which can be treated as another slight contribution of our method to existing memory-augment models and will not cause unfairness in experimental comparisons.\n\nQ4.  More parameters.\n\nA4. Thanks for your valuable advice. Actually, as discussed in Q3, the main body (encoder and decoder) of the network structures used in VSM-VAE and VSM-Diffusion are still the same as their corresponding baselines, and the only additional cost of parameters is brought by introducing the proposed variational structured memory module, which will not bring too much additional memory cost. \nTo make a further investigation, we list the amount of parameters of whole generative models and their memory modules in the following table, and we can find that the proposed variational structure memory will not bring too much increase in the number of parameters.\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| CNS | 7.34|  0%| \n| CNS + semantic| 7.39|  0.68% | \n| CNS + Episodic | 7.38|  0.54%|\n| VSM-CNS | 7.44| 1.36% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| SCHA | 5.37|  0% | \n| SCHA + semantic| 5.60|  +4.28% | \n| SCHA + Episodic | 5.55|  +3.35% |\n| VSM-SCHA| 5.77|  +7.44% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| sDDPM| 34.0|  0% | \n| vDDPM| 35.5|  4.41%| \n| vFSDM | 34.8|  2.35%|\n| VSM-Diffusion| 35.6|  4.70% |\n\nQ5. Source code, how many trials......?\n\nA5. Thanks for your notification. As shown in the supplemental material, we have included the shell files to reproduce our results released on the paper. And we promise that we will absolutely release a more formal version of source code on the github, equipped with a readme file to guide the reproducibility of our work. To make a fair comparison, for each experiment, we take five trials and report the average of these obtained results.\n\n[1] Variational memory addressing in generative models.[2]Variational memory encoder-decoder.[3] Memory transformer.\n[4] Recurrent memory transformer.[5]Learning to learn using gradient descent.[6] Meta-learning with memory-augmented neural networks\n[7]Learning to learn variational semantic memory.[8] Retrieval-augmented diffusion models.[9] Scha-vae: Hierarchical context aggregation for a few-shot generation.[10] Few-shot diffusion models.\n"}, {"review_id": "TUUE8REMaC", "reviewer": "Reviewer_xiFB", "summary": "This work is focused on the problem of few-shot generation using a memory mechanism. A few priors works have leveraged semantic memory or short-term (episodic) memory for such tasks. In this work, a new architecture is proposed named Variational Structured Memory (VSM) that leverages both types of memory mechanisms. Two variants are proposed: VSM-VAE and VSM-Diffusion. The paper discusses the training objective used to train the model and also present ways for recall (using attention and kNNs) from the memory when a new task is available. Lastly, the paper discusses how the memory is updated and initialized. Experiments conducted on existing datasets show that the method outperforms the baselines. Furthermore, the paper also reports the generated images and study the effect of memory size and the memory update process. ", "questions": "1. Semantic Memory Update: The target for the mean of the semantic memory vector seems to favor examples that are close to the current vector of semantic memory using the g(.) function. Why not simply just move the semantic memory towards the mean of all values in vector $H_n$? \n2. Adding to the previous question, the update will favor examples that are closer to the current value of the semantic memory. Will this lead to a mode collapse where the model is unable to generate diverse samples for a category? Is there a way to validate if the model has diversity in the generated samples?\n3. From my understanding, the number of semantic memory modules is the number of categories (N). What if we need to have a different size of semantic memory than the category? Will the model still work?  \n4. Will the method work for out-of-distribution datasets during evaluation because the memories are updated by having higher weights for examples closer to the current memory?\n5. At line 113, should Eq. 5 be referred to instead of Eq. 9?\n6. Minor Typos:\n - Line 135- space between ‘themechanism’\n - Line 151- ‘distribution’ spelling is incorrect\n\n", "limitations": "Limitations are not discussed in the paper. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed idea of using both types of memory for recall is interesting.\n2. The experiments are thorough and support the claims made in the paper.\n3. The paper is well-written and easy to follow. \n", "weaknesses": "1. Some design choices of the algorithm can be explained better (more on this in Questions below). \n2. Limitations of the current method are not discussed. The discussion should include what can be other strategies for updating semantic and episodic memories. The paper should also talk about how to use such an architecture for generating images that can be out of distribution for the new dataset.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and suggestions, which are helpful for us to further improve the quality of our paper. The concerns have been addressed as below.\n\n**W1**: Some design choices of the algorithm can be explained better.\n\n**A1**: Thanks, we will explain your mentioned questions one by one in the following response.\n\n**W2**: Limitations of the current method are not discussed.\n\n**A2**: Thanks for your insightful suggestions. We have discussed other strategies for updating semantic and episodic memories in the response to Q1, and claimed the setting of OOD generation in the response to Q4. We will included these discussions in our revision.\n\n**Q1**: Semantic Memory Update: The target for the mean of the semantic memory vector seems to favor examples...... Why not simply just move the semantic memory towards the mean of all values in vector?\n\n**A1**: Thanks for your valuable suggestions. Actually, we used to consider to utilize the average of the vectors stored in episodic memory to update the semantic memory, but we assume that introducing attention mechanism can highlight the contributions of typical data samples for better model performance. For instance, a typical data sample can serve as the semantic memory of a specific category in the corresponding memory block. To verify our thoughts, we conduct the following experiments to compare these two memory update methods, and the experiment results shown in the table demonstrate the superior of the attention mechanism. Similar conclusions can be found in [1].\n\n| | omniglot | |double-minst ||mnist |   | \n| ----- | -------|  ------- | --------- | --------- |   --------- |  --------- | \n|Update method |  ELBO| NLL| ELBO | NLL | ELBO| NLL |\n| Mean |      77.7 | 60.1 | 46.0| 39.5| 93.8| 70.3|\n| Ours |   74.0 |56.8| 42.4| 36.5| 89.9| 68.2|\n\n**Q2**: the update will favor examples that are closer to the current value of the semantic memory. Will this lead to a mode collapse where the model is unable to generate diverse samples for a category? Is there a way to validate if the model has diversity in the generated samples?\n\n**A2**: We note that the latent representations of data samples retrieved from our memory module only serves as a prior for the downstream generation task, and wouldn't lead to the phenomena of **mode collapse**. \nFor the qualitative results shown in Fig.3, for a novel unseen generation task, we can find that the visualized data samples generated by our model are diverse. Moreover, the metric precision (p) in our experiments can be used to measure the performance of diversity [2]. From the results in the following table, we can find that our developed VSM-based models can outperform on the metirc of precision than other baselines. \n\n| Update method| FID  |SFID | P |  R | \n| ----- | -------|  ------- | --------- | --------- |  \n| Base |    13.52| 27.75| 0.71| 0.38|\n| Mean |    12.33| 25.82| 0.72| 0.40|\n| GAT |   12.27 |25.79| 0.72| 0.40| \n\n[1] Zhen X, Du Y, Xiong H, et al. Learning to learn variational semantic memory[J]. Advances in Neural Information Processing Systems, 2020, 33: 9122-9134.\n\n[2] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.\n\n**Q3**: What if we need to have a different size of semantic memory than the category? Will the model still work?\n\n**A3**: It's an interesting question. Actually, each vector in the semantic memory module can be treated as the clustering center of the corresponding memory block, and the optimal setting of the clustering number should be the number of categories (N), which is also used in [1].  However, it should also work for these memory-augmented models to set less memory blocks, where some similar categories can be summarized with a single memory block. We conduct the following experiment to verify our thoughts as shown in the following table, and the experimental results demonstrate that our model can still work by setting the semantic memory size smaller than the number of categories.\n\n|proportions|Omniglot||Double-mnist||Mnist||\n| ----- | -------|  ------- | --------- | --------- | -------|  ------- | \n| |ELBO| NLL |ELBO| NLL| ELBO| NLL|\n|SCHA + Episodic| 82.5| 63.4| 53| 47.2| 95.3| 79.1|\n|0.1% |81.8| 62.7 |52.2| 46.9 |95.1| 78.6|\n|10% |78.7 |61.5 |46.0 |39.3 |94.6| 72.5|\n|30% |78.6 |60.2 |46.1 |40.2 |94.2 |71.8|\n|50% |76.4| 60.8 |46.0 |39.7 |92.3| 71.9|\n|70% |77.0 |60.2 |44.5 |39.4 |92.2 |71.6|\n|90% |74.6 |57.5| 43.3 |36.4| 90.7 |69.1|\n|100% |74.0 |56.8|42.4 |36.5| 89.9 |68.2|\n\n**Q4**: Will the method work for out-of-distribution datasets during evaluation because the memories are updated with higher weights for examples closer to the current memory?\n\n**A4**: Thanks for your question. Actually, in the setting of few-shot generation, the evaluation is conducted on out-of-distribution datasets, which has been described in Section 3.1. Generally speaking, the generative models will be firstly trained with a series of old tasks (data samples of those seen classes) in the training stage and tested on new tasks (data samples from the other unseen classes). For example, the data samples of old tasks consist of birds, tigers, leopards, and elephants, while the data samples of new tasks will include wolves, dogs and other unseen animals. Thus, in the setting of few-shot generation, the model can learn how to generate OOD data samples by utilizing data information stored in old tasks.\n\n**Q5**: At line 113, should Eq. 5 be referred to instead of Eq. 9?\n\n**A5**. Yes, thanks for your careful verify. We will delete Eq.9 in our revised manuscript.\n\n**Q6** Minor Typos:\nLine 135-  ‘themechanism’\nLine 151- ‘distribution’ \n\n**A6**: Thanks. We will fixed these typos in our revision.\n\nThanks again for your valuable advices on improving our paper, and we hope to take more discussions with you to improve this work.\n\n"}, {"review_id": "WVTL2cslLU", "reviewer": "Reviewer_cAj4", "summary": "The paper proposes a variational structured memory module (VSM) that can be combined with both VAE and Diffusion type models to improve their performance on few-shot generation tasks. The memory structure is both hierarchical and structured into episodic and semantic memory components such that the model can learn to retain detailed experiences as well as higher level general information useful for the task.", "questions": "- It would be interesting to see whether the approach also scales to more challenging datasets such as ImageNet. Are the limitations purely computational or would you expect other challenges?\n- Why is a top-K retrieval strategy used when attention could be applied over all available elements instead? Based on the hyperparameters in the supplementary, the size of the memory (even including episodic memory) does not seem prohibitively large. How would the results change without retrieval, or as you change K?", "limitations": "- How well does the method scale to larger model sizes and/or dataset sizes?\n- The influence of the number of few-shot examples is not discussed. What are the challenges as the number of examples decreases/increases?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- VSM is technically sound and builds on a hierarchical conditioning of latent variables and an attention-structure to retrieve information at inference time.\n- The experiments show that VSM can be combined with existing few-shot generation models (both VAE and Diffusion type models) to improve their performance making it a modular addition widely applicable.\n- An ablation study shows efficacy of the memory types and update strategy", "weaknesses": "- The computational overhead of applying VSM is not discussed. Especially VSM-VAE contains many autoregressive conditionals that would have to be evaluated in sequence. How does the inference runtime compare to not applying VSM?\n- How sensitive is the optimization procedure of VSM to hyperparameters in comparison to not using VSM? Intuitively, the addition of ELBO terms for the memory components could make optimization trickier. Is this observed in practice? How do you deal with this added complexity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions, which are helpful for us to further improve the quality of our paper. The concerns have been addressed as below.\n\n**W1**: Concern on computational overhead of applying VSM.\n\n**A1**:  Thanks for your constructive suggestion. We have provided the additional inference time caused by introducing VSM in the following table. From the results, we can find that the developed VSM-based methods will add no more than 10$\\%$ to the inference time compared to these baselines without VSM, which is quite slight in our understanding.\n\n| Models | Inference time(s)/(Batch_size=32)|  | \n| ----- | -------|  ------- | \n|CNS| | 2.77| 0 |\n|CNS + Semantic | 2.81 | +0.14 |\n|CNS + Episodic | 3.37 | +0.60|\n|VSM-CNS | 3.0924| + 0.32 | \n\n| Models | Inference time(s)/(Batch_size=32)| Rate | \n| ----- | -------|  ------- | \n|SCHA-VAE | 4.17| 0 | \n|SCHA + Semantic | 4.32 | +0.15 |\n|SCHA + Episodic | 6.47 | +2.30 | \n|VSM-SCHA | 4.50| + 0.33  |\n\n| Models | Inference time(s)/(Batch_size=32)| Rate | \n| ----- | -------|  ------- | \n|sDDPM | 98.52 | 0|\n|vDDPM |  92.23 | -5.67|\n|vFSDM | 103.98 | +5.46|\n|VSM-Diffusion | 107.59 | +9.07| \n\n**W2**: Concern hyperparameters in comparison to not using VSM? The addition of ELBO terms for the memory components could make optimization trickier. Is this observed in practice? How do you deal with this added complexity?\n\n**A2**: The optimization of VSM-based models is stable and doesn't face any challenges. Actually, the selection of hyperparameters won't heavily influence the performance of our method. For the hyperparameter $\\alpha$ in the mechanism of memory update, we have conducted the experiment to elaborate on the impact of $\\alpha$, which won't influence the model performance too much as shown in the following table.\n\n|Model| $\\alpha$ | Onmignot| | Double-mnist| |Mnist | |\n|------|-------|------|-------|------|-------|------|------|\n| | | ELBO | NLL | ELBO | NLL| ELBO| NLL|\n|VSM-CNS |0.1 |83.1| 69.5| 49.3| 42.7| 93.2| 77.6|\n|VSM-CNS | 0.5| 83.2 |69.9 |48.1 |42.3 |93.5 |77.8|\n|VSM-CNS |0.7| 81.1 |68.2| 47.8| 40.6| 92.9| 77.0|\n|VSM-CNS |0.9 |84.1 |70.8 |48.7 |42.6 |93.9 |78.5|\n|VSM-SCHA | 0.1 |74.6 |57.0 |42.4 |35.6 |89.7 |68.1|\n|VSM-SCHA |0.5| 73.4| 56.2| 42.2| 35.3| 88.4 |67.1|\n|VSM-SCHA |0.7 |74.0| 56.8 |42.4| 36.5| 89.9 |68.2|\n|VSM-SCHA |0.9 |74.8| 57.5| 43.8| 37.9| 90.1 |69.7|\n\nFor the other hyperparameters in generative models, we directly follow the default hyperparameter settings of baselines to make a fair comparison.\n\n**Q1**: Scales to more challenging datasets such as ImageNet.\n\n**A1**: For scaling our methods to more challenging datasets such as ImageNet, in our undstanding, the computation cost would be the only challenge. As you can see, it will be extremely time-consuming to evaluate our method on the dataset in such scale. Moreover, to improve the generative performance, we may need to increase the number of blocks in the developed structured memory module. \n**We have provided some initial results on miniimagenet in our response to all reviwers** and will try to include more results on imagenet in the revision.\n\n**Q2**: Why is a top-K retrieval strategy used when attention could be applied over all available elements instead? ... How would the results change without retrieval?\n\n**A2**: Actually, the top-K retrieval strategy can be seen as an approach to making the attention weight more sparse, which is effective at alleviating overfitting in attention models [3].  Then, top-K retrieval strategies [5] can also speed up the training, distilling noises without losing information, and increasing the model performance.  Indeed, top-K retrieval strategies have been wildly applied in various memory-augment models [1, 2]. \n\nTo evaluate the effectiveness of top-K retrieval strategy, we provide additional experimental results with various settings of K on Omniglot dataset.\n\n|top-K proportion | Onmignot|  | Double-mnist| |Mnist |   | Training time (mini-batch)|\n| ----- | -------|  ------- |   ----- | -------|  ------- |  --------- | --------- | \n|-- | ELBO|NLL| ELBO|NLL| ELBO|NLL| -- |\n|1%| 73.8| 55.4| 43.2| 36.8| 89.3| 68.5| 79.3s|\n|50%| 77.2| 59.9| 46.8 |39.6| 92.2| 73.8| 86.4s|\n|100% |78.3 |59.6| 47.9 |41.5| 92.2 |76.7 |95.2s|\n\n[1] Zhen X, Du Y, Xiong H, et al. Learning to learn variational semantic memory[J]. Advances in Neural Information Processing Systems, 2020, 33: 9122-9134.\n\n[2] Blattmann A, Rombach R, Oktay K, et al. Retrieval-augmented diffusion models[J]. Advances in Neural Information Processing Systems, 2022, 35: 15309-15324.\n\n[3] Fan, Xinjie, et al. \"Bayesian attention modules.\" Advances in Neural Information Processing Systems 33 (2020): 16362-16376.\n\n[4] Wang, Pichao, et al. \"Kvt: k-nn attention for boosting vision transformers.\" European conference on computer vision\n\n**L1**:How well does the method scale to larger model sizes and/or dataset sizes?\n\n**A1**: We have discussed our method scale to larger dataset sizes in the response to Q1. As for larger model sizes, we believe that the high-level idea of our method can be flexibly applied on large-scale model, like Stable Diffusion, but there will be a lot of technical details needed to be considered in the process of implementation. We will try to explore this direction to introduce memory module into large-scale foundation models.\n\n**L2**: What are the challenges as the number of examples decreases/increases?\n\n**A6**: Actually, there exists a previous work [5] that has discussed the influence of the number of examples decreases/increases to few-show generation. Generally speaking, providing more data samples of a new task can help the generative model to understand the statistical properties of dataset more comprehensively, and further improve the performance of few-show generation.\n\n[5] Giannone G, Winther O. Scha-vae: Hierarchical context aggregation for few-shot generation[C]//International Conference on Machine Learning. PMLR, 2022: 7550-7569."}, {"review_id": "qSzCE7EGVY", "reviewer": "Reviewer_jSnX", "summary": "This paper introduces a novel approach to few-shot generation inspired by the human memory mechanism. The authors propose a Variational Structured Memory module (VSM) to store and recall episodic and semantic memories. They also introduce a bionic memory updating strategy to model the conversion between these memory types. The effectiveness of this approach is demonstrated through its integration with various existing generative models and evaluation of few-shot generation tasks.", "questions": "(1) Can the authors elaborate on how the parameter $\\alpha$ impacts the model's performance? \n\n(2) Is it feasible to apply your method to discriminative models, such as for few-shot classification tasks? Expanding the discussion to these applications would provide a broader view of your method's potential.\n\n(3) Could the authors provide more detailed information regarding the additional model parameters introduced by your method compared to the baseline? Additionally, how does the inference time compare to the baseline? This information could provide more clarity regarding the computational efficiency of your method.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n**Originality**:\nThe paper presents a highly original approach by mimicking the human memory mechanism in the context of few-shot generation. The introduction of the Variational Structured Memory module (VSM) and the bionic memory updating strategy represents a creative combination of cognitive science principles with AI models.\n\n**Quality**:\nThe paper is of high quality, reflected in the meticulous design of the VSM and its integration into existing generative models. The execution of the bionic memory updating strategy and its evaluation through few-shot generation tasks further attest to the paper's quality.\n\n**Clarity**:\nThe paper is well-written. The authors have clearly explained their inspiration, the design of the VSM, and the results of their evaluation, making the paper accessible to readers of varying familiarity with the subject matter.\n\n**Significance**:\nThe significance of this paper is considerable. By demonstrating a novel approach to the few-shot generation and opening up new directions for integrating memory mechanisms into generative models, this work contributes valuable insights to both cognitive science and artificial intelligence fields.\n\n\n\n\n", "weaknesses": "**Computation Efficiency**:\nAlthough the integration of episodic and semantic memory in achieving few-shot generation is intriguing, the use of memory modules inherently increases the computational overhead in terms of extra storage space and prediction time. It would be valuable if the paper could address and analyze these aspects of computation efficiency.\n\n**Acronym Overlap**:\nThe use of the acronym VSM for Variational Structured Memory might confuse, as it has been previously used in the same context in a paper by Zhen et al. [18]. It would be beneficial to consider using a different acronym to avoid any potential confusion for readers.\n\n**Missing Reference and Cross-Domain adaptation**:\nIndeed, the paper has overlooked the reference to the Hierarchical Variational Memory model (HVM) [1].  I would be happy if the authors could discuss whether their approach could be applied to utilize memories from different neural network layers for achieving cross-domain few-shot generation. This discussion could provide additional depth and broaden the applicability of your method.\n\n[1] Du et al. \"Hierarchical variational memory for few-shot learning across domains.\" ICLR 22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and suggestions, which are helpful for us to improve the quality of our paper further. The concerns have been addressed below.\n\nW1: Concern about Computation Efficiency\n\nA1: Thanks, we will answer this question in our response to Q3.\n\nW2: Acronym Overlap:\n\nA2: Thanks for your valuable suggestion. We will update the acronym of our developed method from VSM to VSMM (Variational Structured Memory Module) in the revision. \n\nW3: Missing Reference and Cross-Domain adaptation\n\nA3: Thanks for bringing this excellent work into our view, and we will cite it and discuss its relevance to our work in the revision.For cross-domain adaptation, we believe that our memory-augmented method can be naturally extended for few-shot cross-domain generation task, and the most straightforward solution could be simply replacing the generative model for few-shot generation with another one for few-shot cross-domain generation as the backbone. We are willing to explore this promising direction in the furture.\n\nQ1: Can the authors elaborate on how the parameter  $\\alpha$  impacts the model's performance?\n\nA1: Thanks for your suggestion, as exhibited in the following table, we have conducted the experiment to elaborate on the impact of the hyper-parameter $\\alpha$, and the experimental results show that the setting of parameter $\\alpha$ will not heavily influence the performance of our method. \n\n| Model    | $\\alpha$ |     Onmignot |      | Double-mnist |      | Mnist |      |\n| -------- | -------- | -------- | ---- | ------------ | ---- | ----- | ---- |\n|          |          | ELBO | NLL  | ELBO | NLL  | ELBO  | NLL  |\n| VSM-CNS  | 0.1      | 83.1     | 69.5 | 49.3         | 42.7 | 93.2  | 77.6 |\n| VSM-CNS  | 0.5      | 83.2     | 69.9 | 48.1         | 42.3 | 93.5  | 77.8 |\n| VSM-CNS  | 0.7      | 81.1     | 68.2 | 47.8         | 40.6 | 92.9  | 77.0 |\n| VSM-CNS  | 0.9      | 84.1     | 70.8 | 48.7         | 42.6 | 93.9  | 78.5 |\n| VSM-SCHA | 0.1      | 74.6     | 57.0 | 42.4         | 35.6 | 89.7  | 68.1 |\n| VSM-SCHA | 0.5      | 73.4     | 56.2 | 42.2         | 35.3 | 88.4  | 67.1 |\n| VSM-SCHA | 0.7      | 74.0     | 56.8 | 42.4         | 36.5 | 89.9  | 68.2 |\n| VSM-SCHA | 0.9      | 74.8     | 57.5 | 43.8         | 37.9 | 90.1  | 69.7 |\n\nQ2: Is it feasible to apply your method to discriminative models, such as for few-shot classification tasks? Expanding the discussion to these applications would provide a broader view of your method's potential.\n\nA2: Thanks for your insightful suggestion! Actually, as your mentioned, the proposed variational structured memory module can be naturally extended from few-shot generation to other few-show discrimination tasks by simply replacing the generative models with other discriminative ones. And we believe that our method can also improve the performance of few-shot classification tasks, and will include this part of discussion about the method's potentials in our revision. Thanks again for your reminder.  \n\n\nQ3: Could the authors provide more detailed information regarding the additional model parameters introduced by your method compared to the baseline? Additionally, how does the inference time compare to the baseline? This information could provide more clarity regarding the computational efficiency of your method.\n\nA3: Thanks for your suggestion, we have listed the amount of additional model parameters brought by introducing our method in the following table.\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| CNS | 7.34|  0%| \n| CNS + semantic| 7.39|  0.68% | \n| CNS + Episodic | 7.38|  0.54%|\n| VSM-CNS | 7.44| 1.36% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| SCHA | 5.37|  0% | \n| SCHA + semantic| 5.60|  +4.28% | \n| SCHA + Episodic | 5.55|  +3.35% |\n| VSM-SCHA| 5.77|  +7.44% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| sDDPM| 34.0|  0% | \n| vDDPM| 35.5|  4.41%| \n| vFSDM | 34.8|  2.35%|\n| VSM-Diffusion| 35.6|  4.70% |\n\nWe have also provided the additional inference time caused by introducing our method in the following table.\n\n| Models | Inference time(s)/(Batch_size=32)| Rate |\n| ----- | -------|  ------- |\n|CNS| | 2.77|\n|CNS + Semantic | 2.81 | +0.14 |\n|CNS + Episodic | 3.37 | +0.60|\n|VSM-CNS | 3.0924| +0.32 |\n\n| Models | Inference time(s)/(Batch_size=32)| Rate | \n| ----- | -------|  ------- | \n|SCHA-VAE | 4.17| 0 | \n|SCHA + Semantic | 4.32 | +0.15 |\n|SCHA + Episodic | 6.47 | +2.30 | \n|VSM-SCHA | 4.50| +0.33  |\n\n| Models | Inference time(s)/(Batch_size=32)| Rate | \n| ----- | -------|  ------- | \n|sDDPM | 98.52 | 0|\n|vDDPM |  92.23 | -5.67|\n|vFSDM | 103.98 | +5.46|\n|VSM-Diffusion | 107.59 | +9.07| \n\nWe appreciate your insightful suggestions on how to improve our paper once more, and we look forward to having more conversations with you about this."}, {"review_id": "yTI1H2St9B", "reviewer": "Reviewer_aZnV", "summary": "In this article the authors propose to augment state of the art few-shot generative models with a memory module inspired by the brain. The memory module combines episodic and semantic memory. The authors demonstrate through extensive experimentations the benefits of their module on few-shot generation: when taken either separately or combined, each type of memory improves the generation performance.", "questions": "* In the experimental section (section 5), the authors are comparing the CNS (and the SCHA-VAE) with their model with different type of memory. This comparison is very interesting, but have you controlled for the number of parameters ? On my understanding, there are more parameters in the VSM-CNS (and in the VSM-SCHA) than in the CNS (and SCHA) which makes the comparison not as informative as it should be. Would it be possible to find a way to equate the number of parameters in both type of networks ? \n\n* Concerning the writing : some sentence are just not properly formed. Sometimes, a noun is used instead of a verbs, which make the reading really hard. For exemple : \"which can simultaneously store both episodic and semantic memories to assistant existing generative models efficiently recall memories during generation. » (Line 8-9). In the core text of the article these kind of examples are numerous ! Tenses are also misused, and typos are all over the article !! Please re-read. \n\nHere are few exemples : \nLine 24-25 : A word is missing ?? \\\nLine 113 : Eq 9 —> Eq 5 \\\nLine 135 : TheMechanism —> The Mechanism \\ \nLine 190 : initialed —> initialized \\\n...", "limitations": "The authors have correctly addressed the limitation part", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The claims of the article are well supported by extensive experimentations. Experimentations involves various and diversified datasets, and the baselines are well chosen. The memory module is well described with the Fig2.", "weaknesses": "The writing of the paper makes it unpleasant to read (despite interesting results), and should be improved: some sentences have just not meaning ! I suggest the authors to take time for an ‘in detail’ re-reading to improve the writing (I have just reported few typos, but there are plenty of them !! ). \n\nAs a general comment, the presented article seems to be scientifically correct, but I have quickly lost track of the message because the reading flow was constantly interrupted by a poor writing ! Really sad !", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments and suggestions, which will be helpful for us to further improve the quality of our paper. Following your comments, we have carefully fixed these mentioned typos and also rewritten several parts of our paper for improving the experience of reading.\n\nQ1: Concern on model's parameters.\n\nA1: Yes, it is correct that introducing the proposed variational structured memory module will bring additional model parameters, leading to the amount of parameters of VSM-CNS/VSM-SCHA will be slightly larger than those of CNS/SCHA, respectively. However, we need to highlight that the amount of parameters brought by introducing the proposed variational structure memory is limited, compared to the amount of parameters of the original baselines, which has been listed in the following table.\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| CNS | 7.34|  0%| \n| CNS + semantic| 7.39|  0.68% | \n| CNS + Episodic | 7.38|  0.54%|\n| VSM-CNS | 7.44| 1.36% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| SCHA | 5.37|  0% | \n| SCHA + semantic| 5.60|  +4.28% | \n| SCHA + Episodic | 5.55|  +3.35% |\n| VSM-SCHA| 5.77|  +7.44% |\n\n| Models | Parameters (M)| Rate |\n| ----- | -------|  ------- | \n| sDDPM| 34.0|  0% | \n| vDDPM| 35.5|  4.41%| \n| vFSDM | 34.8|  2.35%|\n| VSM-Diffusion| 35.6|  4.70% |\n\nQ2: Concern on writing.\nA2: Thanks for your valuable suggestions. Following your comments, we have carefully revised our paper.\n\nFor Line 8-9:\nWe have modified Line 8-9 as ``Inspired by the memory mechanism of human brain, in this work, we carefully design a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to assist existing generative models efficiently recall these memories during sample generation``\n\nFor Line 24-25 : A word is missing ??\nWe have modified Line 24-25 as ``While there have been promising processes for few-shot adaptation on classification tasks, less work has been done on few-shot generation [7–10], which is mainly due to the challenging nature of learning the generative process with only a few samples in an unsupervised manner [11, 12].``\n\nFor Eq.9->Eq.5:\nThanks for the notification, we have deleted Eq.9 in the manuscript.\n\nFor typos:\nWe have fixed these listed typos and carefully revised our paper.\n\nWe appreciate your thoughtful comments on our paper once more, and we look forward to having more conversations with you."}]

Author rebuttal:
We really appreciate all the reviewers for their constructive and helpful comments, which can greatly help us to improve the quality of our paper. For the mentioned typos, grammar mistakes, unclear notations and missing citations, we promise that we will carefully fix them  in our revised paper. For all reviewers' questions and suggestions, we have responded one by one in the blow, and you can find it in the corresponding block.



Due to the page limitation, we put a small part of the supplementary experimental results of few-show generation that training the generative models on FScifar100 and then testing them on MiniImageNet in this block. 


| model              | **FID**   | SFID      | P        | R        |
| ------------------ | --------- | --------- | -------- | -------- |
| DDPM               | 63.13     | 33.23     | 0.61     | 0.30     |
| sDDPM              | 47.73     | 32.90     | 0.56     | 0.37     |
| vFSDM              | 61.57     | 32.65     | 0.59     | 0.31     |
| FSDM               | 42.32     | 25.74     | 0.61     | **0.37** |
| VSM-Diffusion(our) | **38.57** | **24.80** | **0.65** | 0.36     |


Meta-review: All reviewers reached a consensus to accept the paper. The authors are encouraged to incorporate all the feedback and suggestions from the reviewers into the final version of the manuscript. Moreover, the authors are also strongly encouraged to benchmark the methods on more complex datasets, such as mini-ImageNet.

After internal discussions, we also recommended the authors to revise the title. "like human being" seems to be a big claim as it lacks solid evidence to support this point explicitly in the paper. Changing to "brain-inspired" or some sort would be more appropriate.

**An important note from the SAC:**
We value the technical strength of the paper, but have some critical concerns, which we ask the authors to address. We decided not to recommend to reject the paper because we believe the authors will respect our note, and follow the changes we ask. We hope our trust is justified. Beyond issues raised by the reviewers and AC in previous discussions, the following issues came up in discussion recently and are critical (but also easy) to address:

1. The title makes an unjustified claim that the approach follows the cognitive process of "human beings". This is not substantiated with any finding from neuroscience/cogsci. As far as we know, this is likely patently incorrect. We ask the authors to change the title and remove any reference to human processes from the paper. It's OK to say in the discussion that you are inspired by human reasoning, while providing references from neuroscience/cogsci. But the claim currently reflected in the title is simply misleading.

2. We ask the authors to remove any reference to their approach as "bionic", as it's currently described in the abstract. Bionic is defined as:
> having artificial body parts, especially electromechanical ones.
This does not apply to the approach. Please remove this description and any similar description from the paper.

Especially in the current days of AI hype, it's critical not to make such claims without the strongest proof possible. It will be irresponsible to see such claim at NeurIPS, one of our field's top venues. 

Thanks,

The SAC
Decision: Accept (poster)