Title: Few-shot Generation via Recalling  Brain-Inspired Episodic-Semantic Memory
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative Model

Memory-augmented Generative Model
Abstract: Aimed at adapting a generative model to a novel generation task with only a few given data samples, the capability of few-shot generation is crucial for many real-world applications with limited data, \emph{e.g.}, artistic domains.
Instead of training from scratch, recent works tend to leverage the prior knowledge stored in previous datasets, which is quite similar to the memory mechanism of human intelligence, but few of these works directly imitate the memory-recall mechanism that humans make good use of in accomplishing creative tasks,  \emph{e.g.}, painting and writing.
Inspired by the memory mechanism of human brain, in this work, we carefully design a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to assist existing generative models efficiently recall these memories during sample generation.
Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories, which can also model the uncertainty during conversion.
Then, we combine the developed VSM with various generative models under the Bayesian framework, and evaluate these memory-augmented generative models with few-shot generation tasks, demonstrating the effectiveness of our methods.

Few-shot Generation via Recalling Brain-Inspired Episodic-Semantic Memory

Abstract

Aimed at adapting a generative model to a novel generation task with only a few given data samples, the capability of few-shot generation is crucial for many realworld applications with limited data, e.g. , artistic domains. Instead of training from scratch, recent works tend to leverage the prior knowledge stored in previous datasets, which is quite similar to the memory mechanism of human intelligence, but few of these works directly imitate the memory-recall mechanism that humans make good use of in accomplishing creative tasks, e.g. , painting and writing. Inspired by the memory mechanism of human brain, in this work, we carefully design a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to assist existing generative models efficiently recall these memories during sample generation. Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories, which can also model the uncertainty during conversion. Then, we combine the developed VSM with various generative models under the Bayesian framework, and evaluate these memory-augmented generative models with few-shot generation tasks, demonstrating the effectiveness of our methods.

1 Introduction

A remarkable capability of human intelligence is its ability to quickly grasp the concepts of new objects that it has never encountered before [1]. Essentially, this rapid adaptation is achieved by utilizing the past memories stored in the human brain, which can greatly improve the efficiency of later learning [2, 3]. Unlike human brain, existing machine learning methods often require to be trained on a large amount of data when facing new tasks, motivating the research interests on few-shot learning that aim at efficiently solving these unseen tasks with only a few given data samples [4-6]. While there have been promising processes for few-shot adaptation on classification tasks, less work has been done on few-shot generation [7-10], which is mainly due to the challenging nature of learning the generative process with only a few samples in an unsupervised manner [11, 12].

For existing deep neural networks, to improve their abilities of few-shot adaptation, one of the most effective methods is to introduce an augmented memory module, which can store the memory of past experiences to adapt the model to a new task quickly [6, 13-16]. However, the most widely used memory mechanism in exisiting works is short-term [17] with limited capacity, which focuses on the

Mingyuan Zhou

collection of personal experience related to a particular place or time. To remove the limitation of short-term memory, Zhen et al. [18] introduce the concept of semantic memory, which allows the storage of general conceptual information into meta-learning to acquire long-term knowledge for few-shot learning. Although these memory modules have separately achieved attractive progress in few-shot learning [1, 2, 19, 20], there is little work to explore a memory mechanism to organically combine the semantic and episodic memories, and further apply it on few-shot generation tasks.

Inspired by the memory mechanism of the human brain, where both episodic and semantic memory work together to help humans understand the world around us and also accomplish creative tasks, in this work, we first develop a variational structured memory module (VSM), which can simultaneously store both episodic and semantic memories to improve the few-shot generation capability of the generative models. Then, we design two important processes in VSM: 1) structured memory recall, which retrieves the relevant information of previous tasks from VSM to be applied on current task; 2) structured memory update, which continuously collects data information from training tasks and gradually consolidates the structured memories stored in VSM. Finally, we combine VSM with a branch of few-shot latent variable models, specifically neural statistician (NS) [7], and develop a series of memory-augmented latent variable models for few-shot generation.

The main contributions of this paper can be summarized as follows:

Inspired by the memory mechanisms of the human brain, we develope a novel variational structured memory module (VSM) that can concurrently store both episodic and semantic memories, effectively aiding generative models in memory recall during sample generation.

Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories stored in VSM, where episodic memory can provide detailed episodes that will be converted later and stored in semantic memory.

Through incorporating VSM with NS, we develop a novel hierarchical memory-augmented neural statistician (HMNS), whose generative process can be specified with V AE or diffusion model, leading to VSM-VAE and VSM-Diffusion, respectively.

Extensive experiments demonstrate that our developed VSM-VAE and VSM-Diffusion can achieve promising generative performance on few-shot generation tasks.

2 Related work

Few-Shot Latent Variable Models: For few-shot generation, a series of latent variable models based on NS [7] have been developed [9-12], which can be further improved by adopting a non-parametric formulation for the context [21], introducing a hierarchical learnable aggregation for the input set [11], increasing expressivity for the conditional prior [10], or exploring supervision [22]. Specifically, NS [7] introduces a context constant variable to specify the summary statistics of a dataset, which provides a shared prior for these samples within the same dataset, contributing to handling new datasets at test time; GMN [9] defines a variational model to learn a per-sample context-aware latent variable. Distinct from previous works, our method introduces a memory module to collect historical datasest information to inference the context of a new dataset.

Memory-Augmented Neural Networks: It has been shown that augmenting existing neural networks with an external memory module can improve their capability of few-shot learning [23]. Specifically, these memory-augmented neural networks mainly collect the information stored in the support set of the current task [6, 15, 16], which will be wiped form episode to episode [17] instead of maintaining long-term information that has been shown to be crucial for effectively learning new unseen tasks. For instance, variational semantic memory [18] is proposed to accumulate and store the semantic information from previous tasks for the inference of the prototypes of new tasks. Distinct from previous methods only constructing semantic memory for the prototype network, in this paper, we develop a structured memory module to store the historical information at both the episodic and semantic sides, which can largely enhance the model's capability of few-shot generation.

i

c

i

c

i



i

i

i



i

i

c



c

1,...,

1,...,

M



1,...,

1,...,

M

M

M

j

z

i

,

j

z

i

x

z

,

j

,

i

i

z

x

i

j

,

x

i



j

j

,

,

j

j

,

j

,

j

i

i

x



1,...,

1,...,

j

j

1,...,

1,...,

N



N



N

N

m

i

m

i

m

1,...,

i

j

J







1

i

m





1

,

i

j

z





,

L

i

j

z





1

l

i

m







l

i

m

1,...,

i

j

J



,

i

j

x





1

i

c

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z





l

i

c





1

l

i

c







L

i

c

1,...,

i

j

J



,

i

j

x

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z

i

c





L

i

m





L

i

c





1

i

c





2

i

c





1

l

i

c



,

i

j

x





2

,

i

j

z





1

,

l

i

j

z



1,...,

i

j

J



,

i

j

x

1,

,

i

N



1,

,

i

N



i

m

i

c





1

,

i

j

x





,

l

i

j

x





1

,

l

i

j

x







,

L

i

j

x

(a) NS

1,...,

i

j

J







1

i

m





1

,

i

j

z





,

L

i

j

z





1

l

i

m







l

i

m

1,...,

i

j

J



,

i

j

x





1

i

c

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z





l

i

c





1

l

i

c







L

i

c

1,...,

i

j

J



,

i

j

x

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z

i

c





L

i

m





L

i

c





1

i

c





2

i

c





1

l

i

c



,

i

j

x





2

,

i

j

z





1

,

l

i

j

z



1,...,

i

j

J



,

i

j

x

1,

,

i

N



1,

,

i

N



i

m

i

c





1

,

i

j

x





,

l

i

j

x





1

,

l

i

j

x







,

L

i

j

x

(b) HNS

1,...,

i

j

J







1

i

m





1

,

i

j

z





,

L

i

j

z





1

l

i

m







l

i

m

1,...,

i

j

J



,

i

j

x





1

i

c

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z





l

i

c





1

l

i

c







L

i

c

1,...,

i

j

J



,

i

j

x

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z

i

c





L

i

m





L

i

c





1

i

c





2

i

c





1

l

i

c



,

i

j

x





2

,

i

j

z





1

,

l

i

j

z



1,...,

i

j

J



,

i

j

x

1,

,

i

N



1,

,

i

N



i

m

i

c





1

,

i

j

x





,

l

i

j

x





1

,

l

i

j

x







,

L

i

j

x

(c) VSM-VAE

1,...,

i

j

J







1

i

m





1

,

i

j

z





,

L

i

j

z





1

l

i

m







l

i

m

1,...,

i

j

J



,

i

j

x





1

i

c

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z





l

i

c





1

l

i

c







L

i

c

1,...,

i

j

J



,

i

j

x

1,...,

i

N







1

,

i

j

z





,

l

i

j

z





1

,

l

i

j

z







,

L

i

j

z

i

c





L

i

m





L

i

c





1

i

c





2

i

c





1

l

i

c



,

i

j

x





2

,

i

j

z





1

,

l

i

j

z



1,...,

i

j

J



,

i

j

x

1,

,

i

N



1,

,

i

N



i

m

i

c





1

,

i

j

x





,

l

i

j

x





1

,

l

i

j

x







,

L

i

j

x

(d) VSM-Diffusion

i

m

3 Memory-Augmented Neural Statistician

3.1 Preliminary



z



L





L

i

c



i

c

c





1

i

i

l

l









1



1

i

  1 , i j z   , L i j z   L i c   1 c   2 i c   1 l i c    2 , i j z   1 , l i j z    1 , i j z   , L i j z   L i c   1 i c   2 i c   1 l i c    2 , i j z   1 , l i j z    1 , i j z , i j i c   1 i c   2 i c   1 l i c    2 , i j z   1 , l i j z    1 , i j z   , L i j z   L i c   1 i c   2 i c   1 l i c    2 , i j z   1 , l i j z  Few-shot Generation : Few-shot generation [22] borrows a similar setting of few-shot learning [4] but for a more challenging task to generate objects given only a few samples of that object at test time. Specifically, given a set of training datasets D old = { D i } N i =1 , where each dataset D i = { x ij } J i j =1 consists of J i data samples, the goal of few-shot generation is to train a generative model p θ ( x ) that is trained on given datasets D old but can be quickly adapted to a new unseen task given only a few data samples collected from a new dataset D new.

x , i j x , i j x , i j x 3.2 Memory-Augmented Neural Statistician

,

j

j



1,...,

N

i

1,..., j N  1,..., j N  1,..., j N  Through incorporating memory mechanism with few-shot latent variable models, specifically NS [7] in Figure 1(a), in this paper, we develop a novel memory-augmented NS (MNS) for few-shot generation.

Generative Process of MNS: The design of MNS aims at introducing a memory module to store and update the data information at various semantic levels, which can be recalled when facing a new generative task. As shown in Figure 1(c), the generative process of hierarchical MNS is formulated as:

To clarify the meanings of z i,j , c i , and m i : z i,j denotes the per-sample latent variables of the j th sample in dataset D i , c i denotes the task-specific context variable of dataset D i , whose detailed definition can be found in [7], and m i denotes the task-specific memory variable, which stores the data information of previous tasks that will be adapted to the new task.

3.3 Variants of Hierarchical Memory-Augmented Neural Statistician

Through specifying the generative process of HMNS and replacing its memory module with the variational structured memory (VSM) as described in Section 4, we can obtain a series of model



c

l



1



l



m

i

i

m

m

i

i

c

i

c

i

c

c

i

i

M

M





M

M

M

L



L



M



M

l





l





1

1

l

L



M

M

M





l





l

M

L







1

l





M





1

M



1



M











l





1



l

1







1





M

M

m





i

i

m

m



m



m

L

L







i





l



l

i

i

m

m

m

1

1

L

L





i





l





i

m



l



l

i

i

m

m

i







i

m



1

l

1



l





l



i

1

1





i



1



i



i



i

m

m

m

m

variants, such as VSM-HVAE and VSM-Diffusion, both of which can achieve promising generative performance on few-shot generation task.

3.3.1 VSM-VAE

Generative Process of VSM-VAE: To capture the hierarchical data information at various semantic levels, we extend the latent variables in Eq. (1) to a hierarchical version, specifically z i,j = { z ( l ) i,j } L l =1 , c i,j = { c ( l ) i,j } L l =1 , and m i,j = { m ( l ) i,j } L l =1 , which will be included in a variational autoencoder (V AE) with L hidden layers, leading to a VSM-VAE shown in Figure 1(c). Under the generative framework of MNS, using the same symbol notation in Eq. (1), the per-sample generative process of x i,j in the dataset D i can be formulated as

where p θ ( z i,j | c i ) , p θ ( c i | m i ) , and p ( m i ) are formulated as the product of corresponding hierarchical latent variables as follows

Training Objective of VSM-VAE: For VSM-VAE, the training objective is to maximize the evidence lower bound (ELBO) of a set of training datasets { D i } N i =1 , which can be expressed as

The first item in Eq. (5) can be further written as

and Q denotes the variational distribution defined by the inference network of VSM-V AE

where M indicates the memory module to store the semantic information collected from previous tasks, and will be introduced in Section 4.

3.3.2 VSM-Diffusion

Generative Model of VSM-Diffusion: Further, through specifying the generative process of HMNS with the diffusion model, we can obtain the VSM-Diffusion, as shown in Figure 1(d). Distinct from the hierarchy of latent variables in VSM-V AE, we share the same context and memory variables across all hidden layers of a diffusion model, specifically c i = c ( l ) i and m i = m ( l ) i for l ∈ { 1 , ..., L } . Thus, under the generative formulate of MNS, using the same symbol notation in Eq. (1), the per-sample generative process of x i,j in the dataset D i can be formulated as

where the backward diffusion process is consistent with that of conditional diffusion model [24, 25], formulated as

Training objective of VSM-Diffusion: For VSM-Diffusion, the training objective is derived from maximizing the evidence lower bound (ELBO) of a set of training datasets { D i } N i =1 , which is consistent with Eq. (5). Similar to regular DDPMs, the first term can be decomposed into a summation of L terms, each of which can be computed independently, and can be estimated as

where L L is a fixed constant, L 0 is the expectation of likelihood term and the intermediate term L l -1 can be formulated as

4 Variational Structured Memory

In this section, we will introduce the storage architecture of VSM in Section 4.1, equipped with the mechanism of memory recall and update in Sections 4.2 and 4.3, respectively. The basic intuition of VSM is inspired by the memory mechanism in the human brain: 1) there are two advanced forms of memory in the human brain: semantic memory allows the storage of general conceptual information and episodic memory allows the collection of detailed episodes; 2) episodic memory can provide detailed episodes that will be converted into conceptual information and stored in semantic memory. The structured memories stored in VSM will be recalled to infer the posterior of task-specific memory, formulated as:

which will be used in the inference process of either VSM-VAE or VSM-Diffusion.

4.1 Structured Memory Storage

As shown in Figure 7, there are N blocks (categories) in the VSM module, denoted as M = { M n } N n =1 , and each memory block M n consists of both semantic memory M ( s ) n and episodic memory M ( e ) n .

Semantic Memory Storage: The semantic memory in the human brain can provide conceptual information for quickly learning concepts of new object categories by seeing only a few data samples. Following Zhen et al. [18], for each block (category) of semantic memory in VSM, we try to store the conceptual information by averaging the latent representations of data samples belonging to this category, contributing to lightweight storage and fast lookup. To model the uncertainty, we model the n -th block (category) of semantic memory with a Gaussian distribution, formulated as:

where µ n , σ n ∈ R d denotes the d -dimensional mean and variance vectors, respectively.

Episodic Memory Storage: Distinct from semantic memory allowing the storage of general conceptual information, episodic memory tends to focus on storing more detailed and vivid experiences or episodes. The design of episodic memory is inspired by previous works [26-30], and the main idea is to employ a query memory module to collect a subset of representative data samples for each block (category) in VSM. Specifically, the n -th block (category) of episodic memory is a set of T embedding vectors of data samples:

where T is the storage limit of each block (category) of episodic memory, and the dimension of M ( e ) n,t ∈ R d is the same as the latent representations of data samples.











1

M

2

M

3

M

M

N

Set Feature

Episodic

Memory

K nearest neighbor lookup

Semantic

Memory

(

)

M

s

d

n

∈



(

)

,

M

e

d

n t

∈



(

)

s

d

i

m

∈



(Key, Value)

(Query)

Semantic

Memory

d

i

h

∈



(

)

e

d

i

m

∈



Episodic

Memory

,1

i

x

,2

i

x

,3

i

x



KNN Attention

4.2 Structured Memory Recall

Given a query dataset D i = { x i,j } J i j =1 , which can be treated as a new dataset D new for previous tasks, the mechanism of memory recall aims at effectively using the memory module to quickly adapt a model trained on previous tasks to a new unseen task. The workflow of structured memory recall in VSM has been depicted in Figure 7, which includes both the recalls of semantic memory M ( s ) and episodic memory M ( e ) , resulting in two kinds of task-specific memories, denoted as m ( s ) i and m ( e ) i respectively.

Semantic Memory Recall: For a new few-shot generation task on the dataset D i ( D new ), the memory recall in VSM will first select the top-K relevant blocks (categories) of semantic memory as the candidates, denoted as { M ( s ) k } K k =1 , and then use an attention mechanism to aggregate these candidates into a task-specific semantic memory m ( s ) i , where the attention weight of n -th can be calculated as

where g ( · ) can be treated as a similarity function, and h i indicates the semantic representation of dataset D i and can be obtained by averaging the latent representations of data samples in D i .

Then, with the top-K relevant blocks of semantic memory { M ( s ) k } K k =1 in hand, we can further aggregate these candidates with their attention weights, formulated as:

where m ( s ) i ∈ R d has gathered the relevant information from the semantic memory M ( s ) for the generation task on dataset D i .

Episodic Memory Recall: After selecting the top-K relevant blocks of semantic memory { M ( s ) k } K k =1 , we can further dive into recalling episodic memory, which is more specific and detailed. In practice, each block of episodic memory contains T embedding vectors of data samples M ( e ) k = { M ( e ) k,t } T t =1 , and we can finally obtain K ∗ T embedding vectors of data samples. After that, following a similar process of recalling episodic memory, we can further aggregate these obtained embedding vectors with their attention weights as

where g ( · ) and h i have been defined in Eq. (13), and the obtained m ( e ) i ∈ R d has gathered the relevant information from the episodic memory M ( e ) for the generation task on dataset D i .

4.3 Structured Memory Update and Consolidation

Semantic Memory Update: Inspired by the characteristics of human memory [31-33], we develop a novel semantic memory update mechanism that considers the data information from previous semantic memory M ( s ) , episodic memory M ( e ) , and new task D i . Specifically, the semantic memory module is empty at the beginning of learning, and each unseen block (category) M ( s ) n will be initialized with the mean and variance of the latent representations of data samples, which is collected from the current task D i and belongs to n -th category. For those seen categories, their semantic memory can be updated with the data sample from the current task D i using self-attention mechanism.

Following Zhen et al. [18], we construct a graph with respect to the memory M ( s ) n for updating, which consists of latent representations of J n data samples that belongs to n -th category and T data samples stored in episodic memory M ( e ) n , denoted as H n = { h n, 1 , h n, 2 , · · · , h n,J n , M ( e ) n, 1 , · · · , M ( e ) n,T } . The mean of the semantic memory M ( s ) n can be updated as:

where α ∈ (0 , 1) is a hyperparameter; β n,j indicates the attention weight between j -th node features and the source node M ( s ) n ; g ( · ) is the similarity function defined in Eq. (13). This operation allows the useful information to be retained in memory while erasing less relevant or trivial information.

After that, we employ the incremental update strategy of the Gaussian distribution [34] to update the variance of semantic memory M ( s ) n as follows:

where ¯ H n and σ 2 H n are the mean and variance of the set of node features H n , respectively.

Episodic Memory Update: Inspired by the fact that it is difficult for a person to forget those things he/she often recalls (and vice versa), we design a frequency-based episodic memory update stragy. Typically, for each data sample M ( e ) n,t stored in episodic memory M ( e ) n , we use a frequency matrix, denoted as C M ( e ) n,t , to record how many times M ( e ) n,t was recalled during training. Then, for updating M ( e ) n = { M ( e ) n,t } T t =1 , if the current episodic memory block has not been filled yet, we will directly append the latent representation of a new data sample to the M ( e ) n when a new task arrives; Otherwise, we will update M ( e ) n by replacing the least used data sample in the block if its capability exceeds the limit T . We note that the frequency matrix C M ( e ) n,t will be reset as the zero matrix after each update of episodic memory. The detailed memory updating process algorithm can be found in Appendix A.

5 Experiments

5.1 Experimental setup

Datasets: The experiments are conducted on five widely used benchmark datasets of various sizes, including binarized Omniglot [2], MNIST [35], DOUBLE-MNIST [36], CelebA [37] and FS-CIFAR100 [38]. The split of training/testing set (also known as background-evaluation split) follows Lake et al. [2], in which scenario all the samples at test time are from new classes. We resize all the images of binary datasets to 28x28, FS-Cifar100 to 32x32, and CelebA to 64x64.

Baselines: 1) For VAE-based models, we utilize NS-related models, specifically focusing on two NS variants: Convolutional Neural Statistician (CNS), where the latent space is shaped with convolutions at a given resolution; Set-Context-Hierarchical-Aggregation-Variational-Autoencoder (SCHAVAE) [24], which employs an additional hierarchy over the context latent variable c . The CNS and SCHA-VAE can be naturally extended under the framework of VSM-VAE, leading to VSM-CNS and VSM-SCHA, respectively. Meanwhile, we create several variants by fusing either semantic memory or episodic memory with CNS or SCHA-VAE, as appropriate, to further demonstrate the efficacy of the proposed variational structured memory; 2) For Diffusion-based models, we consider both

unconditional and conditional diffusion models as baselines, where we use the DDPM [39] as the unconditional baseline and other three conditional diffusion models: VDDPM, where the context c is a latent variable and the dataset D is generated conditioned on c ; VFSDM [24], another variational diffusion model where employees a different way to extract and aggregate set-information using ViT [40]: stack all the samples on the channel dimension; sDDPM adapting ideas in Sinha et al. [41] without contrastive learning, where a ViT encoder splits the image in patches and processes them jointly.

Model Settings: The details about model settings can be found in Appendix B.

5.2 Quantitative experiment

VAE-based models : To evaluate the generative properties of V AE-based models, we use the evidence lower bound (ELBO) to approximate the log marginal likelihood with 1000 importance samples (NLL). Our qualitative experiments are performed on Omniglot, DOUBLE-MNIST, MNIST, and FS-Cifar100. The outcomes of the experiment are displayed in Table 5.2. It is clear that VSMCNS/VSM-SCHA can achieve significant improvements over its baseline models, proving that VSM could be a promising mechanism for few-shot generation. Additionally, by incorporating either semantic or episodic memory into CNS/SCHA-VAE, the resulting models can also achieve better performance, proving both memory types can improve the model performance of few-shot generation.

Diffusion-based models: As exhibited in Table 5, we employ the metrics below to assess the generative capacity of diffusion-based models, including: FID [42] for sample quality, sFID [43] to capture spatial relationships, and Precision and Recall [44] for measuring variety and mode coverage. From the results, VSM-Diffusion beats the other unconditional and conditional baselines on both two datasets, demonstrating the effectiveness of VSM on few-shot generation.

Set

、 阿;

』

`｀

，

_.....T

`

set

semantic memory

episode memory

Samples

set set

semantic memory

(b) Recalled Semantic Memory episode memory

episode memory

0

1

3

5

10

Episodic memory size

50

55

60

65

70

75

80

85

90

NELBO / NLL

NELBO

NLL

(a) Omniglot

0

1

3

5

10

Episodic memory size

50

55

60

65

70

75

80

85

90

NELBO / NLL

NELBO

NLL

(b) Doublemnist

0

1

3

5

10

Episodic memory size

60

70

80

90

100

110

NELBO / NLL

NELBO

NLL

(c) Mnist

5.3 Qualitative experiment

Few-shot image generation: The generation results of VSM-Diffusion are exhibited in Figure 3. From the results, we can find that our memory-augmented models can generate class-specific samples through successfully extracting content information and realistic classes that are both complicated and varied. Limited by pages, additional few-shot generative experimental results on Omniglot, CelebA, and FS-Cifar100 can be found in Appendix C.2.

Memory visualization: To further verify the effectiveness of the proposed VSM, we conduct experiments of memory visualization here. For instance, given a new set, as shown in Figure 4(a), the images of recalled semantic memory corresponding to the class are shown in Figure 4(b), and the images of recalled episode memory are shown in Figure 4(c). The visualization results demonstrate that VSM can recall related image classes and samples to the query set in an effective way.

5.4 Ablation study

Effect of memory size: To investigate the affect of memory size in episodic memory, we use the VSM-SCHA model as a typical example. We train a model on Omniglot data, and evaluate it with the Omniglot, DOUBLE-MNIST, and MNIST datasets for unknown classes. The size of the episodic memory is raised from 0 to 10. The results in Figure 5 show that the model performance improves with the increase of episodic memory size, further demonstrating the importance of episodic memory in few-shot generation tasks. We note that the developed variational structured memory will reduce to variational semantic memory [18] when the episodic memory size is adjusted to 0.

Effect of memory update process: We conduct two variants of VSM-SCHA to investigate the benefits of the proposed structured memory update strategy. For the first variant, its semantic memory update strategy does not consider episodic memory, denoted as W/o Episodic memory ; For the second variant, its episodic memory update strategy does not employ a frequency-based strategy but uses a first-in, first-out queue-based memory update method, denoted as W/o Frequency-based . The experiment results in Table 3 can verify the effectiveness of the proposed memory update strategy.

6 Conclusion

In this study, we develop a novel variational structured memory module that can concurrently retain generic semantic structure about a category and specific information about a sample, the former being achieved by semantic memory and the latter being caught by episodic memory. Meanwhile, we introduce a bionic memory updating strategy for the conversion between episodic and semantic memories stored in VSM, where episodic memory can provide context information that will later be transferred and stored in semantic memory. Through incorporating VSM with few-shot latent variable models, we develop a novel hierarchical memory-augmented neural statistician (HMNS), whose generative process can be specified with a V AE or diffusion model, leading to VSM-V AE and VSMDiffusion. Extensive experiments demonstrate that our developed VSM-VAE and VSM-Diffusion can achieve promising performance on few-shot generation tasks.

Acknowledgments

This project was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22160; in part by the 111 Project under Grant B18039; in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSJ23016; Additionally, this project was supported in part by the National Research Foundation, Singapore under its Industry Alignment Fund - Pre-positioning (IAF-PP) Funding Initiative.

References

Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society , volume 33, 2011.

Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science , 350(6266):1332-1338, 2015.

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical bayes. arXiv preprint arXiv:1801.08930 , 2018.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Harrison Edwards and Amos Storkey. Towards a neural statistician. arXiv preprint arXiv:1606.02185 , 2016.

Danilo Rezende, Ivo Danihelka, Karol Gregor, Daan Wierstra, et al. One-shot generalization in deep generative models. In International conference on machine learning , pages 1521-1529. PMLR, 2016.

Sergey Bartunov and Dmitry Vetrov. Few-shot generative modelling with generative matching networks. In International Conference on Artificial Intelligence and Statistics , pages 670-678. PMLR, 2018.

Luke B Hewitt, Maxwell I Nye, Andreea Gane, Tommi Jaakkola, and Joshua B Tenenbaum. The variational homoencoder: Learning to learn high capacity generative models from few examples. arXiv preprint arXiv:1807.08919 , 2018.

Giorgio Giannone and Ole Winther. Scha-vae: Hierarchical context aggregation for few-shot generation. In International Conference on Machine Learning , pages 7550-7569. PMLR, 2022.

Zhibin Duan, Yishi Xu, Jianqiao Sun, Bo Chen, Wenchao Chen, Chaojie Wang, and Mingyuan Zhou. Bayesian deep embedding topic meta-learner. In International Conference on Machine Learning , pages 5659-5670. PMLR, 2022.

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850. PMLR, 2016.

Łukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. arXiv preprint arXiv:1703.03129 , 2017.

Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International conference on machine learning , pages 2554-2563. PMLR, 2017.

Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In International conference on machine learning , pages 3664-3673. PMLR, 2018.

Augusto Buchweitz. Models of working memory: Mechanisms of active maintenance and executive control (1999). Ilha do Desterro A Journal of English Language, Literatures in English and Cultural Studies , (43):193-200, 2002.

Xiantong Zhen, Yingjun Du, Huan Xiong, Qiang Qiu, Cees Snoek, and Ling Shao. Learning to learn variational semantic memory. Advances in Neural Information Processing Systems , 33: 9122-9134, 2020.

Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences , 40, 2017.

Tomer D Ullman and Joshua B Tenenbaum. Bayesian models of conceptual development: Learning as building models of the world. 2020.

Mike Wu, Kristy Choi, Noah Goodman, and Stefano Ermon. Meta-amortized variational inference and learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 6404-6412, 2020.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8): 1735-1780, 1997.

Giorgio Giannone, Didrik Nielsen, and Ole Winther. Few-shot diffusion models. arXiv preprint arXiv:2205.15463 , 2022.

Andreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller, and Björn Ommer. Retrievalaugmented diffusion models. Advances in Neural Information Processing Systems , 35:1530915324, 2022.

Arslan Chaudhry, Marc'Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420 , 2018.

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems , 30, 2017.

Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. Advances in neural information processing systems , 32, 2019.

Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning. Advances in Neural Information Processing Systems , 33: 1023-1035, 2020.

Jun He, Richang Hong, Xueliang Liu, Mingliang Xu, Zheng-Jun Zha, and Meng Wang. Memoryaugmented relation network for few-shot learning. In Proceedings of the 28th ACM International Conference on Multimedia , pages 1236-1244, 2020.

Endel Tulving. Episodic and semantic memory. 1972.

Larry R Squire and Stuart M Zola. Episodic memory, semantic memory, and amnesia. Hippocampus , 8(3):205-211, 1998.

Daniel L Greenberg and Mieke Verfaellie. Interdependence of episodic and semantic memory: Evidence from neuropsychology. Journal of the International Neuropsychological society , 16 (5):748-753, 2010.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Yann LeCun, Lawrence D Jackel, Léon Bottou, Corinna Cortes, John S Denker, Harris Drucker, Isabelle Guyon, Urs A Muller, Eduard Sackinger, Patrice Simard, et al. Learning algorithms for classification: A comparison on handwritten digit recognition. Neural networks: the statistical mechanics perspective , 261(276):2, 1995.

Shao-Hua Sun. Multi-digit mnist for few-shot learning, 2019. URL https://github.com/ shaohua0116/MultiDigitMNIST .

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision , pages 3730-3738, 2015.

Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems , 31, 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems , 34:12533-12548, 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint arXiv:2103.03841 , 2021.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems , 32, 2019.

A Memory Recall and Update Algorithm

The procedure of memory recall and update algorithm is summarized in Algorithm. 1.

Algorithm 1 Semantic Memory recall and update algorithm

Input : Given task D i ; Semantic memory M ( s ) , and episodic memory M ( e ) ; Recall task-specific semantic memory according to Eq. 14 and episodic memory according to Eq. 15 ; if The semantic memory for the given task is null: then Computing the mean and variance for the semantic memory with given task D i ; else Update the semantic memory according to Eq. 16 and Eq. 17; end if if The episodic memory for the given task is null: then Append episodic memory with given task D i ; else Update the episodic memory according to frequency matrix C ; end if Infer variational posteriors for the latent variables of different layers { θ ( l ) } L 1=1 ;

Model Design: In most memory-based models, the images D i = { x ij } J i j =1 are fed into an encoder to produce image features { h ij } J i j =1 ∈ R D × 1 , which are then represented as vectors. These vector representations are utilized for both recall and update operations in the model. Nonetheless, due to the high dimensionality of images, relying solely on vectors can lead to a loss of valuable information.By drawing inspiration from the approach proposed in the Vision Transformer, we can overcome these challenges and obtain more accurate memory information. We begin by obtaining image features, denoted by { h ij } J i j =1 ∈ R D × H × W , then calculate the average of the set dimension resulting in h i = 1 J i ∑ J i j =1 h ij Afterwards, we reshape the size of the image features to { h i } ∈ R N × D where D denotes the patch size.When updating M ( s ) n ∈ R N × D , we employ the resized feature h i m ∈ R ( J i × N ) × D for graph attention. As for M ( e ) n ∈ R L × N × D , where L denotes the total number of episodic memory, we directly replace the least recently used patch in M ( e ) n with h i m .By using per-patch recall and update, we obtain information from different semantic levels. This makes the Semantic Memory more suitable for task generalization, while the Episodic Memory is more capable of capturing sample-specific details.

VSM-SCHA In SCHA-VAE and VSM-SCHA, we use a shared routing path between c and z: q ϕ ( c ( l ) i | D i , c ( l +1) i , z ( l +1) i , m ( l ) i ) where z l +1 and c l +1 are samples from layer l . We additionally remove any form of normalization (batch/layer norm) in the hierarchical formulation

Memory size Across all datasets, we set the size of each semantic memory block (category) in the VSM-based model to 1, and each episodic memory block to 5. For models that exclusively depend on episodic memory (CNS + Episodic, SCHA + Episodic, and Episodic-diffusion), we establish a memory size of 10000, with the episodic memory functioning as a first-in, first-out queue.

B Details of Model Design

C Additional Experiment Results

C.1 Quantitative Comparison of Generative Samples

In this section, we apply VSM to FSDM, a conditional diffusion model where c is not a variational variable. To align with FSDM, we only use the following update rule for the semantic memory:

Input

Encoder Block_1

32

×

️

32

Encoder Block_2

16

×

16

8

×

8

Encoder Block_4

4

×

4

2

×

2

Encoder Block_6

1

×

1

Semantic memory_1

8

×

8

Episodic memory_1

8

×

8

Semantic memory_2

4

×

4

Episodic memory_2

4

×

4

Semantic Block_3

2

×

2

Episodic Block_3

2

×

2

Semantic Block_4

1

×

1

Episodic Block_4

1

×

1

Inference Block_c_1

8

×

8

Inference Block_c_2

4

×

4

Inference Block_c_3

2

×

2

Inference Block_c_4

1

×

1

Inference Block_z_1

8

×

8

Inference Block_z_2

4

×

4

Inference Block_z_3

2

×

2

Inference Block_z_4

1

×

1

Output

Decoder Block_1

32

×

️

32

Decoder Block_2

16

×

16

8

×

8

Decoder Block_4

4

×

4

Decoder Block_5

2

×

2

Decoder Block_6

1

×

1

Input

Encoder_Block

Vit

Input

Semantic memory

Episodic memory

Spatial Transformer

×

3

ResBlock

×

3

Spatial Transformer

×

3

ResBlock

×

3

MiddleBlock

Decoder_Block

Spatial Transformer

×

3

ResBlock

×

3

Spatial Transformer

×

3

ResBlock

×

3

InferenceBlock_c

Output

Table 4 presents the results of VSM-FSDM on Fs-Cifar100 and celeba datasets. It can be observed that VSM performs well even without using variational latent variables, demonstrating its strong performance and scalability.

C.2 Additional Visualization of Generative Samples

More visualization results of generative samples can he found in Figures 8, 9, 10 and 11.

D Border Impact

The impact of our research can be highlighted in two key aspects:

For Few-shot Learning Task: Our structured memory has demonstrated a high level of effectiveness in storing and managing task-related information, enabling efficient transfer of prior knowledge from previous tasks to new ones. By combining semantic and episodic information storage, our model enhances its ability to capture both task-level information and fine-grained image-level details. Diverging from traditional few-shot models, our approach leverages the stored and recalled memory to extract more valuable insights while minimizing the influence of irrelevant information on new tasks.

For Deep Generative Models: Our modularized memory offers a seamless integration into a wide range of generative models, such as diffusion models and VAEs, while also easily merging with latent variables. This flexibility allows for efficient utilization of the memory component within the generative framework. In addition, our structured memory can be divided into multiple semantic levels, providing the ability to learn at various refined semantic layers. This hierarchical organization of memory facilitates capturing complex dependencies and relationships across different levels of

abstraction. By incorporating multiple semantic layers, generative models can potentially achieve enhanced performance and generate more meaningful and coherent outcomes.

E Limitation

The proposed model variant, VSM-Diffusion, is based on the denoise diffusion model, which requires multiple steps to generate samples. The framework can be extended to other diffusion models, such as implicit denoise diffusion models, to accelerate the generative process and improve its efficiency.

GYOOTO

，

今

,...

、

'

..

f

-．

丛．J

a

;气丘·勹勹

阻墅登

. .

I

>

....,...

－

－-

w · 4.

、

四

：、广_

气

-

齐已习之户

I

一人，仅亡

I

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this article the authors propose to augment state of the art few-shot generative models with a memory module inspired by the brain. The memory module combines episodic and semantic memory. The authors demonstrate through extensive experimentations the benefits of their module on few-shot generation: when taken either separately or combined, each type of memory improves the generation performance.\n\nSTRENGTHS:\nThe claims of the article are well supported by extensive experimentations. Experimentations involves various and diversified datasets, and the baselines are well chosen. The memory module is well described with the Fig2.\n\nWEAKNESSES:\nThe writing of the paper makes it unpleasant to read (despite interesting results), and should be improved: some sentences have just not meaning ! I suggest the authors to take time for an ‘in detail’ re-reading to improve the writing (I have just reported few typos, but there are plenty of them !! ). \n\nAs a general comment, the presented article seems to be scientifically correct, but I have quickly lost track of the message because the reading flow was constantly interrupted by a poor writing ! Really sad !\n\nQUESTIONS:\n* In the experimental section (section 5), the authors are comparing the CNS (and the SCHA-VAE) with their model with different type of memory. This comparison is very interesting, but have you controlled for the number of parameters ? On my understanding, there are more parameters in the VSM-CNS (and in the VSM-SCHA) than in the CNS (and SCHA) which makes the comparison not as informative as it should be. Would it be possible to find a way to equate the number of parameters in both type of networks ? \n\n* Concerning the writing : some sentence are just not properly formed. Sometimes, a noun is used instead of a verbs, which make the reading really hard. For exemple : \"which can simultaneously store both episodic and semantic memories to assistant existing generative models efficiently recall memories during generation. » (Line 8-9). In the core text of the article these kind of examples are numerous ! Tenses are also misused, and typos are all over the article !! Please re-read. \n\nHere are few exemples : \nLine 24-25 : A word is missing ?? \\\nLine 113 : Eq 9 —> Eq 5 \\\nLine 135 : TheMechanism —> The Mechanism \\ \nLine 190 : initialed —> initialized \\\n...\n\nLIMITATIONS:\nThe authors have correctly addressed the limitation part", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to few-shot generation inspired by the human memory mechanism. The authors propose a Variational Structured Memory module (VSM) to store and recall episodic and semantic memories. They also introduce a bionic memory updating strategy to model the conversion between these memory types. The effectiveness of this approach is demonstrated through its integration with various existing generative models and evaluation of few-shot generation tasks.\n\nSTRENGTHS:\n**Originality**:\nThe paper presents a highly original approach by mimicking the human memory mechanism in the context of few-shot generation. The introduction of the Variational Structured Memory module (VSM) and the bionic memory updating strategy represents a creative combination of cognitive science principles with AI models.\n\n**Quality**:\nThe paper is of high quality, reflected in the meticulous design of the VSM and its integration into existing generative models. The execution of the bionic memory updating strategy and its evaluation through few-shot generation tasks further attest to the paper's quality.\n\n**Clarity**:\nThe paper is well-written. The authors have clearly explained their inspiration, the design of the VSM, and the results of their evaluation, making the paper accessible to readers of varying familiarity with the subject matter.\n\n**Significance**:\nThe significance of this paper is considerable. By demonstrating a novel approach to the few-shot generation and opening up new directions for integrating memory mechanisms into generative models, this work contributes valuable insights to both cognitive science and artificial intelligence fields.\n\nWEAKNESSES:\n**Computation Efficiency**:\nAlthough the integration of episodic and semantic memory in achieving few-shot generation is intriguing, the use of memory modules inherently increases the computational overhead in terms of extra storage space and prediction time. It would be valuable if the paper could address and analyze these aspects of computation efficiency.\n\n**Acronym Overlap**:\nThe use of the acronym VSM for Variational Structured Memory might confuse, as it has been previously used in the same context in a paper by Zhen et al. [18]. It would be beneficial to consider using a different acronym to avoid any potential confusion for readers.\n\n**Missing Reference and Cross-Domain adaptation**:\nIndeed, the paper has overlooked the reference to the Hierarchical Variational Memory model (HVM) [1].  I would be happy if the authors could discuss whether their approach could be applied to utilize memories from different neural network layers for achieving cross-domain few-shot generation. This discussion could provide additional depth and broaden the applicability of your method.\n\n[1] Du et al. \"Hierarchical variational memory for few-shot learning across domains.\" ICLR 22.\n\nQUESTIONS:\n(1) Can the authors elaborate on how the parameter $\\alpha$ impacts the model's performance? \n\n(2) Is it feasible to apply your method to discriminative models, such as for few-shot classification tasks? Expanding the discussion to these applications would provide a broader view of your method's potential.\n\n(3) Could the authors provide more detailed information regarding the additional model parameters introduced by your method compared to the baseline? Additionally, how does the inference time compare to the baseline? This information could provide more clarity regarding the computational efficiency of your method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a variational structured memory module (VSM) that can be combined with both VAE and Diffusion type models to improve their performance on few-shot generation tasks. The memory structure is both hierarchical and structured into episodic and semantic memory components such that the model can learn to retain detailed experiences as well as higher level general information useful for the task.\n\nSTRENGTHS:\n- VSM is technically sound and builds on a hierarchical conditioning of latent variables and an attention-structure to retrieve information at inference time.\n- The experiments show that VSM can be combined with existing few-shot generation models (both VAE and Diffusion type models) to improve their performance making it a modular addition widely applicable.\n- An ablation study shows efficacy of the memory types and update strategy\n\nWEAKNESSES:\n- The computational overhead of applying VSM is not discussed. Especially VSM-VAE contains many autoregressive conditionals that would have to be evaluated in sequence. How does the inference runtime compare to not applying VSM?\n- How sensitive is the optimization procedure of VSM to hyperparameters in comparison to not using VSM? Intuitively, the addition of ELBO terms for the memory components could make optimization trickier. Is this observed in practice? How do you deal with this added complexity?\n\nQUESTIONS:\n- It would be interesting to see whether the approach also scales to more challenging datasets such as ImageNet. Are the limitations purely computational or would you expect other challenges?\n- Why is a top-K retrieval strategy used when attention could be applied over all available elements instead? Based on the hyperparameters in the supplementary, the size of the memory (even including episodic memory) does not seem prohibitively large. How would the results change without retrieval, or as you change K?\n\nLIMITATIONS:\n- How well does the method scale to larger model sizes and/or dataset sizes?\n- The influence of the number of few-shot examples is not discussed. What are the challenges as the number of examples decreases/increases?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is focused on the problem of few-shot generation using a memory mechanism. A few priors works have leveraged semantic memory or short-term (episodic) memory for such tasks. In this work, a new architecture is proposed named Variational Structured Memory (VSM) that leverages both types of memory mechanisms. Two variants are proposed: VSM-VAE and VSM-Diffusion. The paper discusses the training objective used to train the model and also present ways for recall (using attention and kNNs) from the memory when a new task is available. Lastly, the paper discusses how the memory is updated and initialized. Experiments conducted on existing datasets show that the method outperforms the baselines. Furthermore, the paper also reports the generated images and study the effect of memory size and the memory update process.\n\nSTRENGTHS:\n1. The proposed idea of using both types of memory for recall is interesting.\n2. The experiments are thorough and support the claims made in the paper.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Some design choices of the algorithm can be explained better (more on this in Questions below). \n2. Limitations of the current method are not discussed. The discussion should include what can be other strategies for updating semantic and episodic memories. The paper should also talk about how to use such an architecture for generating images that can be out of distribution for the new dataset.\n\nQUESTIONS:\n1. Semantic Memory Update: The target for the mean of the semantic memory vector seems to favor examples that are close to the current vector of semantic memory using the g(.) function. Why not simply just move the semantic memory towards the mean of all values in vector $H_n$? \n2. Adding to the previous question, the update will favor examples that are closer to the current value of the semantic memory. Will this lead to a mode collapse where the model is unable to generate diverse samples for a category? Is there a way to validate if the model has diversity in the generated samples?\n3. From my understanding, the number of semantic memory modules is the number of categories (N). What if we need to have a different size of semantic memory than the category? Will the model still work?  \n4. Will the method work for out-of-distribution datasets during evaluation because the memories are updated by having higher weights for examples closer to the current memory?\n5. At line 113, should Eq. 5 be referred to instead of Eq. 9?\n6. Minor Typos:\n - Line 135- space between ‘themechanism’\n - Line 151- ‘distribution’ spelling is incorrect\n\nLIMITATIONS:\nLimitations are not discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to incorporate 'episodic' and 'semantic' memory inspired by the human memory system into existing generative modeling frameworks to improve few-shot generation abilities. Specifically, they propose a variational structured memory module (VSM) and show its incorporation into existing VAE based models and diffusion based models can improve few-shot generation across multiple datasets. The memory mechanism is introduced as a prior to the context variables in traditional neural statistician (NS) and diffusion modeling frameworks. The ‘semantic’ memory is modeled as a ‘general’, lightweight storage with quicker retrieval (single embedding for each category) while ‘episodic’ memory is modeled to be more detailed/'vivid' (a set of embeddings for each category) and provide context relevant information. Learning involves an attention-based recall/lookup from prior memory states and subsequent update of the semantic and episodic memory for each category.\n\nSTRENGTHS:\n1. The integration and formulation of proposed memory modules (and relevant storage, recall and update mechanisms) to improve few-shot generation is relatively novel.\n2. The experiments are performed on established datasets and the results support the claim that the proposed VSM module can benefit performance on few-shot generative/diffusion-based tasks.\n3. The method and underlying components are largely adequately described in section 3 and 4 with relevant references or background provided.\n\nWEAKNESSES:\n1. The work's claim of modelling the memory mechanism like a 'human being' for few-shot generation tasks seems far-fetched and not adequately backed by literature or explanation. While episodic and semantic memory indeed are components of human memory (and some references for this are provided), it is not clear how they are related to static image few-shot generation (specifically, episodic memory is largely related to an agent's experiences/events and seems more relevant to episode/reinforcement learning based tasks/frameworks). \n\n2. Further, it remains unclear what is the authors mean when they mention 'context' in relation to the proposed semantic and episodic memory. E.g. on L146, it is mentioned 'semantic memory in human brain can provide context information'; while L139 mentions 'episodic memory can provide context information'. So it is unclear if both semantic and episodic are to provide 'context' and if so what does 'context' mean here? Again, in relation to my first point, backing statements such as L146 with references would be beneficial.\n\n3. Experiment setup: It is unclear whether all models were evaluated with the same vision encoder and processing. (In appendix L401 authors mention they draw inspiration from vision transformer in processing image features and suggest their process differs from existing memory-based models). Were other models evaluated in a similar setup for fair comparison? Otherwise, performance benefits might be due such differences in experimental setup and not necessarily the proposed memory method itself.  \n\n4. Further, authors should consider indicating the parameters of their method and other relevant factors to provide a better comparison to existing models (e.g. see Table1 in SCHA-VAE: Hierarchical Context Aggregation for Few-Shot Generation Giorgio Giannone, Ole Winther International Conference on Machine Learning, ICML, 2022). It is currently unclear whether the performance benefits may merely be due to more parameters or if it is due to the proposed method.\n\n5. While source code is provided in appendix, would it be released publicly for reproducibility? (currently there is no indication). Also, how many trials were performed for each experiment? \n\nMinor:\n1. The writing in introduction has minor grammar mistakes and missing words / wrong phrasing (e.g. L9: 'to assistant' -> 'to assist'; L18: 'that never encountered before' -> 'that it has never encountered before'; L24: 'which mainly' -> 'which is mainly'; L49 'to assistant' -> to assist', etc)\n\nQUESTIONS:\nPlease see weaknesses section above.\n\nLIMITATIONS:\nA brief limitation section is provided in the appendix L449. Perhaps it could be expanded to discuss potential negative applications of generative/diffusion-based methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}