ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
Authors: Qingzhong Ai
LIRONG HE
SHIYU LIU
Zenglin Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep generative model
VAE
Coreset

Abstract: Recent studies show that advanced priors play a major role in deep generative models. Exemplar VAE, as a variant of VAE with an exemplar-based prior, has achieved impressive results. However, due to the nature of model design, an exemplar-based model usually requires vast amounts of data to participate in training, which leads to huge computational complexity. To address this issue, we propose Bayesian Pseudocoresets Exemplar VAE (ByPE-VAE), a new variant of VAE with a prior based on Bayesian pseudocoreset. The proposed prior is conditioned on a small-scale pseudocoreset rather than the whole dataset for reducing the computational cost and avoiding overfitting. Simultaneously, we obtain the optimal pseudocoreset via a stochastic optimization algorithm during VAE training aiming to minimize the Kullback-Leibler divergence between the prior based on the pseudocoreset and that based on the whole dataset. Experimental results show that ByPE-VAE can achieve competitive improvements over the state-of-the-art VAEs in the tasks of density estimation, representation learning, and generative data augmentation. Particularly, on a basic VAE architecture, ByPE-VAE is up to 3 times faster than Exemplar VAE while almost holding the performance. Code is available at \url{https://github.com/Aiqz/ByPE-VAE}.

ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
Qingzhong Ai Lirong He! Shiyu Liu'
Zenglin 23
School of Computer Science and Engineering,
University of Electronic Science and Technology of China, Chengdu China School of Science and Technology,
Harbin Institute of Technology Shenzhen, Shenzhen China
DDeaatment of Network Intelligence,
Peng Cheng National Lab, Shenzhen, China
(qzai,1irong, 1ii.onggheeeeeeeee. @atd. uestc. edu. cn, shyu diuufoxmaai com, xuzenglin@hit, edu.
Abstract
Recent studies show that advanced priors play major role in deep generative models. Exemplar VAE, as a variant of VAE with an exemplar-based prior, has achieved impressive results. However, due the nature of model design, exemplar-based model usually requires vast amounts data to participate in train- ing, which leads huge computational complexity. To address this issue, we propose Bayesian Pseudocoresets Exemplar VAE (ByPE- VAE with prior based on Bayesian pseudocoreset. The proposed prior is condi- tioned on a small-scale pseudocoreset rather than the whole dataset for reducing the computational cost and avoiding overfitting. Simultaneously. we obtain the optimal pseudocoreset via stochastic optimization algorithm during VAE training aiming to minimize the Kullback-Leiblee divergence between the prior based on the pseudocoreset and that based on the whole dataset. Experimental results show that ByPE-VAE VAE can achieve competitive improvements over the sate-of-the-arr art VAEs in the tasks of density estimation, representation learning, and generative data augmentation. Particularly, on a basic VAE architecture, ByPE-VAE is up to 3 times faster than Exemplar VAE while almost holding the performance. Code is available hhttp:::// : //gi thub. com/ www.o/aigg/yy.........o
new variant
Introduction
Deep generative models that learn implicit data distribution from the enormous amount of data have received widespread attention to generating highly realistic new samples machine learning. In particular, due to the utilization of the reparameterization trick and variational inference for optimization, Variational Autoencoders (VAEs) 2 stand out and have demonstrated significant successes for dimension reduction
33, learning representations 4 and generating data (5. In addition, various variants of VAE have been proposed conditioned on advanced variational posterior 7 g or powerful decoders 9 10].
worth noting that the prior the typical VAE is a simple standard normal distribution that convenient to compute while ignores the nature of the data itself. Moreover, large number of experiments have empirically demonstrated that simplistic priors could produce the phenomena of over-regularization and posterior collapse, and finally cause poor performance [5] [T]. Hence, many researchers have worked to develop more complex priors to enhance the capacity of the variational posterior. In this line, Tomezak al. (12) introduces more flexible prior named VampPrior, which is a mixture of variational posteriors based on pseudo-inputs to alleviate the problems like overfitting
CCoresponding Author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

and high computational cost. However, the way in which the pseudo-inputs are obtained is not interpretable. Recently, Norouzi et al. [13 develops an Exemplar VAE with a non-parametric prior based on an exemplar-based method, achieving excellent performance. To ensure the performance and the generation diversity, the exemplar set needs to he large enough, and usually the entire training data is utilized. Obviously, this leads to huge computational complexity. Even though Exemplar VAE further employs approximate nearest neighbor search to speed up the training process, the number of nearest neighbors should be as large possible to ensure performance. In a nutshell, Exemplar VAE computationally expensive.
To address such issues, we develop a new prior for VAE that is inspired by the paradigm of coresets [14]. The coreset is a powerful tool aiming to find small weighted subset for efficiently approximat- ing the entire original dataset. Therefore, rather than using the large-scale training data directly, we seek to design a prior conditioned on coreset, which greatly reduces the computational complexity and prevents overfitting. In practice, to better incorporate this idea with the framework of VAE, we further employ specific form of coresets, namely Bayesian pseudocoresets [1]. which known as small weighted subset of the pseudodata points, resulting in a Bayesian pseudocoreset based prior. With this prior, we gain a new variant of VAE called Bayesian Pseudocoresets Exemplar VAE (ByPE-VAE). To sample from the ByPE-VAE, we first take pseudodata point from the pseudocoreset according to its weight, and then transform it into a latent representation using the learned prior. Then decoder is used to transform the latent representation into sample.
crucial part of ByPE-VAE VAE is to obtain the optimal pseudocoreset. We formulate this process as variational inference problem where the pseudodata points and corresponding weights are the parameters variational posterior approximation. More precisely, to seek the optimal pseudocoreset, we minimize the Kullback-Leibler (KL) divergence between the prior based on the pseudocoreset and that based on the entire dataset. This processing ensures that the learned prior is actually an approximation of the prior conditioned on whole training data. Thus, is fundamentally different from general pseudodata based priors, like VampPrior. For optimization, we adopt two-step alternative search strategy to learn two types of parameters of ByPE-VAE, which refer to the parameters in the VAE framework and the pseudodata points and corresponding weights in the pseudocoreset. In particular, iteratively optimize one of the two types of parameters while keeping the other one fixed until convergence.
Finally, we compare ByPE-VAE with several state-of-the-art VAEs in number of tasks, including density estimation, representation learning and generative data augmentation. Experimental results demonstrate the effectiveness of ByPE-VAE on Dynamic MNIST, Fashion MNIST, CIFARIO 10, and CelebA. Additionally, to validate the efficiency of our model, we measure the running time on basic VAE architecture. Compared to the Exemplar VAE, ByPE- is to 3 times speed-up without losing performance on Dynamic MNIST, Fashion MNIST, and CIFARIO.
Preliminaries
Before presenting the proposed model, we first introduce some preliminaries, namely Exemplar VAE and Bayesian pseudocoresets. Throughout this paper, vectors are denoted by bold lowercase letters, whose subscripts indicate their order, and matrices are denoted by upper-case letters.
2.1 Exemplar VAE
Exemplar VAE is regarded as variant of VAE that integrates the exemplar-based method into VAE for the sake of seeking impressive image generation. Specifically, first draws random exemplar x,, using uniform distribution from the training data X = {x, J.W (where N denotes the sample amount), then transforms an exemplar x,, into latent variable z using an example-based prior (z X,), finally generates observable data x by a decoder ps(x z). The parametric transition distribution (x xn) of deemplar-bssed generative models can be expressed as
(x xn) / ro(zz (z) xx)po(x |z) z))z,
(1)
where & and 0 denote corresponding parameters. Assuming that x is independent to x,, conditioned on the latent variable and marginalizing over the latent variable the objective O(8, of

---

Exemplar VAE can be formulated as
N
1
(6(2|x)
logp(x: X,0, 0) = log
To To.0(x(xn)) 2 (x/x)) log logpo(x(z) - Bao( log Σ rod (2)2))))/
N
= -
(2)
where O00, o; X) known as the evidence lower bound (ELBO). From the Eq. 2 it can be derived that the difference between Exemplar VAE and typical VAE is the definition of the prior p(z) in the second term. The prior of Exemplar VAE defined as a mixture form, that is p(z x) Σx=1 (z xn)/N. The variational posterior (o(z x) and the exemplar-based prior (2 X,, are assumed to be Gaussian distributions whose parameters are fulfilled by neural networks. Note that, the computational cost of this training process is related to the number of exemplars which usually set be the entire training data. This indicates that Exemplar VAE is computationally expensive since the amount of training data is generally huge.
2.2 Bayesian Pseudocoresets
Bayesian pseudocoresets is method of coreset construction based on variational inference, which constructs a weighted set of synthetic "pseudodata" instead of the original dataset during inference. First, the goal of this method is to approximate expectations under the posterior π(b)) with the parameter tt, which is formulated as
N Σ F(em: eb) To(#), n==
1 ( = exp
(3)
where ((, v) denotes a potential function, and usually is log-likelihood function. (o(b)) the prior, and Z is the normalization constant. Instead of directly inferring the posterior π (V²), the Bayesian pseudocoreset employs a weighted set of pseudodata points to approximate the true posterior (v)). which is given by
1
TU. (1b)
Zu.w
exp
Σ Wmf(u
(/) To(V).
(4)
where { Um Im=1 represents M pseudodata points Um Rd, w {um J m=1 denotes non- negative weights, and Zu. is the corresponding normalization constant. Finally, this model obtains the optimal pseudodata points and their weights by minimizing the KI. divergence, as follows,
U w
argmin DKL (TU,w([(T)
(5)
U.w
This formulation can reduce the computational cost by decreasing data redundancy.
Bayesian Pseudocoresets Exemplar VAE
To generate a new observation x, the Exemplar VAE requires a large collection of exemplars from X = =x,,,, [= to guide the whole process, as shown Eq. (2). One can see that the greater the number of exemplars set, the richer the prior information can be obtained. In practice, to ensure performance, the number of exemplars is relatively large which generally set the size of the entire training data. This leads to huge computational costs in the training process of the Exemplar VAE, To overcome such a issue, inspired by the paradigm of Bayesian pseudocoresets, we adopt M pseudodata points U = {um) with corresponding weights
= {wm JM to denote exemplars, importantly M < N. That is, the original exemplars X are approximated by a small weighted set of pseudodata points known Bayesian pseudocoreset. The framework can be expressed as
pox X, 0) log
To xx.) log
Wm To um)) log p(x U, w. O). N
(6)
m==
where Wm 20 I,... M) and /w/l1 = = N.. Further, we integrate this approximated framework with VAE by introducing a latent variable z. Then the parametric function given by
Tx um)
[T((z) [ um um) (x z)dz,
3

---

where (z Um) with parameter & denotes a pseudodata based prior for generating z from pseu- dodata point un- Po(x z) with parameter represents the decoder for generating the observation x from z. Similarly, we assume that an observation is independent from a pseudodata point Um conditional on z to simplify the formulation and optimization.
In general, we desire to maximize the marginal log-likelihoode log p(x) for learning, however, this is intractable since we have no ability to integrate the complex posterior distributions out. Now, we focus on maximizing the evidence lower bound (ELBO) derived by Jensen's 's inequality, as follows,
pog U, w, 0) log Σ
Wm To.o(x Um) log
Σ m=1
Wm
J. ro(z) um um) z)dz (8)
N
20(z x)
E (x/x) log logpo(x) z) (x/xx) log
m=1 Wmrolz um)/N
= = o, U, w)
where (z(z x) represents the approximate posterior distribution. And O o, U, w; x) is defined as the objective function of the ByPE-VAE VAE to optimize parameters 0 and Φ. The specific derivation can be found in Supp[A] As we can see from Eq. ( the difference between the ByPE-VA. VAE and other variants of VAE is the formulation of the prior p(z) in the second term. In detail, the prior of the ByPE- is a weighted mixture model prior, in which each component is conditioned on a pseudodata point and the corresponding weight, i.e., p(z[U, w) = Em=1 Wm To (z(um)/N. In contrast, the prior of the Exemplar VAE is denoted as p(z]X) - ro(z(x.)/N.
As shown in Eq. (z ). And the distributions of (2(2 x) and rz(z | Um) are both designed as Gaussian distributions. According to the analysis in [16). the optimal prior is the form of aggregate posterior. Inspired by this report, make the prior be coupled with the variational posterior, follows,
the ByPE-VAE includes two encoder networks, namely qo x) and
(z x) = [2 Aa(x),Aa(x)) (z um)) N(z ps (um)) (u,)),
(10)
(11)
We employ the same parametric mean function Ps for two encoder networks for better incorporation. However, the covariance functions of the two encoder networks are different. Specifically, the variational posterior uses a diagonal covariance matrix function Aç. while each component of the mixture model prior uses isotropic Gaussian with a scalar parameter o². Note that the scalar parameter o² is shared by each component of the mixture model prior for effective computation. Then, we can express the log of the weighted pseudocoreset based prior log P& (z U, w) as
||z psd (lm))l
logpo(z U, w) = dd log( (2To)) - log + log
Σ 172
Wm exp
(12)
20²
where dz denotes the dimension of z. Based on the formulation of Eq. objective function of the ByPE- VAE, follows,
12 we can further obtain the
O(8, o. U. w: X) (x/(x
log ( z)
log Σ
Wm
exp
||z
pb
(um)
(z x)
(V2mo)
20²
(13)
log (x 2))
M
Wm
1z
1b (um)
!1
E (xx())x.)
log
exp
To (z x;)
(v2
20
(14)
where the constant - log N is omitted for convenience. For (xg(z)x;)) we employ the reparametriza- tion trick to generate samples. Note that, for the standard VAE with a Gaussian prior, the process of generating new observations involves only the decoder network after training. However, to generate
new observation from ByPR-VAN we not only require the decoder network, but also the learned Bayesian pseudocoreset and pseudodata point based prior rs. We summarize the generative process of the ByPE- VAE in Algorithm

---

Algorithm 1: The Generative Process of ByPE-VAE
Input Pseudocoreset {U, w), Decoder pe, Prior Ta.
Output:A generated observation x.
Step.1 Sample um Multi(- U, ( for obtaining pseudodata point um from the learned pseudocoreset.
Step.2 Sample z r.(-] Um using the pseudodata point based prior ra for obtaining the latent
representation
Step.3 Sample x po(- using the decoder po for generating new observation
However, more importantly, the whole process above holds only if the pseudocoreset is an approxima- tion of all exemplars or training data. To ensure this, we first re-represent frameworks of (z X) and Ps (z U, w) from the Bayesian perspective. Concretely, pe (z X) and po U, w) are also viewed as posteriors conditioned on the likelihood function po( z) and a certain prior Pz(z), follows,
N
Pa(z (X) =
exp
Σ W=1
log pe (Xn 2) po(z),
(15)
Z
M
Pz U,
exp
Wm log po Um um
z)) pu(z),
(16)
Zu.w
m=1
where Z and Zu.w are their respective normalization constants, po( 22) here specifically refers the decoder. Then, we develop this problem into a variational inference problem, where the pseudodata points and the corresponding weights the parameters of the variational posterior approximation followed [TS]. Specifically, we construct the pseudocoreset by minimizing the KL divergence in terms of the pseudodata points and the weights, as follows,
U w" argmin DKL (po(z) U, w)llp(z X)). U.w
(17)
with respect to the pseudodata point um and the weights w Further, the gradients of DKL. in Eq. are given by
Vun DKL = Wm Covu,w V log
12), log po po 2 1N - log po z)" 2), lo z) 7 IN log (U 2
(18)
V DKL = Covu.w [log (U
(19)
where Covu,w denotes the covariance operator for the (2 U, w). and 1 E R-W represents the vector of all entries. In practice, adopt black-box stochastic algorithm obtain the optimal pseudodata points and weights. Details of these derivations are provided in Supp B and C
Note that ByPE- VAE involves two types of parameters, namely the parameters 0 and á in the VAE framework and the parameters U and w the pscudocoreset. For optimization, we use a two-step alternative optimization strategy. In detail, (i) update 0 and á with fixed pseudocoreset (U, w) and (ii) update U and w with fixed and Φ. Steps (i) and (ii) are iteratively implemented until convergence. The detailed optimization algorithm shown in Algorithm 2 One
that the computation does not scale with N, but rather with the number of pseudocoreset points M, which greatly reduces the computational complexity and also prevents overfitting. Also note that, rather than be updated every epoch, the pseudocoreset (U, w) } is updated by every h epochs. And k is set to 10 in the experiments.
4 Related Works
Variational Autoencoders (VAEs) 1 are effectively deep generative models that utilize the variational inference and reparameterizztioo trick for dimension reduction [3), learning representations 4], knowledge base completion [17] and generating data [5]. And various variants of VAE have been proposed conditioned on advanced variational posterior [6] g, powerful decoders 9 10 and flexible priors [10
12 13 As for the prior, the standard VAE takes the normal distribution the prior, which may lead the phenomena of over-regularization and posterior collapse and further affect the performance for density estimation (5] [ In the early stage, VAEs apply more complex
18
5

---

Algorithm 2: The Optimization Algorithm for ByPE-VAE
Input
Training data X = {Xn batch size B, training epochs 7,, learning rate "Y. sample size S, pscudocoresct size M, update interval &
Decoder po, variational posterior Ιo. weighted pseudocoreset based prior Pe
Initialized pscudocoreset by which B ~ UnifSubset ([N],)) {b1, ..... bas 1m f- Xhm W/n +- N/M, m = 1,..,,
Output:Parametie :aaametees 0 and d. Pseudocoreset {U. w
for 1,.... ,T' do
/* Optimize VAE parameters 0 and & + --ww++ (NNM w.mean) Centralized to w
Evaluate the ByPE-VAE objective using Eq. (14. and update and & using the ADAM
/+ Optimize pseudocoreset Um and Wm, m = 1, MM
if t.kk = then
Take samples from current pseudocoreset posterior ps (z) U, w). namely (z) ~ Po(z|U, (0, w) Obtain mini-batch of datapoints B ~ UnifSubset ([N], B)
for 1,.... S do
8. (log po (x (z.) 1/5 1/ log logpo(xolz..)
R
g. < (log (oogoo(uu (um k.) 1/5 1/5 -og Loggooumm (..m)mm2))))))))))))) m=m
E R
for 1, ,,., M do
h a +- V log (m (2.) 1/5 Vo log (a,))) E Rd
11
12
V. f- 1/8 Σs g. N/ 8.
13
for m = 1, ,..,, do
V um
Wmn ES
fim,s (N/Bg." 5. w)
15
W t max w Vw.
16
for m 1,... M do
Um Um Vt Um
priors, such the Dirichlet process prior [19], the Chinese Restaurant Process prior (20), to improve the capacity of the variational posterior. However, these methods can only be trained with specific tricks and learning methods. Chen al. (T0] employs the autoregressive prior which is then along with convolutional encoder and an autoregressive decoder to ensure the performance of generation.
Tomezak et al. 122 introduces variational mixture of posteriors prior (VampPrior) conditioned set of pseudo-inputs aiming at approximating the aggregated posterior. The intent of our method is similar to the VampPrior with respect to the of pseudo-inputs. Nevertheless, there is fundamental difference. The pscudo-inputs in [12] are regarded as hyperparameters of the prior and are obtained through backpropagation, while the pseudo-inputs of our model are the Bayesian pseudocoreset and are optimized through variational inference. Therefore, the pseudo-inputs learned by our model could approximate all the original data, with the weighting operation carried on. In other words, our model is casier to understand in termss of interpretability.
Exemplar VAE [13 is variant of VAE with non-parametric prior based method to learn desirable hidden representations. Exemplar VAE takes all training data to its exemplar set instead of pseudo-inputs. This computational cost is expensive since the amount of training data can be huge. Hence, Exemplar VAE further presents the approximate kNN search to reduce the cost. However, this technique could reduce the effectiveness of the algorithm and is used only in the training process. Our model introduces the Bayesian pseudocoreset under the Exemplar VAE framework, which improves only the computational speed, but also the performance of VABs.
seemplar-based
In addition, a eeemory-augmente generative model with the discrete latent variable (1 is proposed to improve generative models. Our model can be considered a VAE with additional memory. There are two essential differences between our model and [2T]. First, the pscudo-inpur are based on the Bayesian pseudocoreset which is easy to interpret. Second, model doesn't it need a normalized categorical distribution.

---

Experiments
Experimental setup We evaluate the ByPE- VAE on four datasets across several tasks based on multiple network architectures. Specifically, the tasks involve density estimation, representation learning and data augmentation, the used four datasets include MNIST, Pashion-MNIST, CIFAR 10, and CelebA, respectively. Following [13]. for the first three datasets, we conduct experiments on three different VAE architectures, namely VAE based MLP with two hidden layers, HVAE based on MLP with two stochastic layers, and a ConvHVAE based on CNN with two stochastic layers. Following (22), adopt the convolutional architecture for CelebA. In addition, we measure the running time on the first network architecture for three datasets. The ADAM algorithm with 5e-4. And use KL annealing for 100 epochs and carly-stopping with a look ahead of 50 epochs. In addition, the weights of the neural networks are initialized according to [26). Following [12] and [13), we use Importance Weighted Autoencoders (IWAE) [27] with 5000 samples for density estimation.
normalized gradients [23 24 25 is used for optimization and learning rate is
5.1 Density Estimation
To validate the effectiveness of SyPE-VAE, we compare ByPE- with state-of-the methods for each architecture, namely a Gaussian prior, VampPrior, and an Exemplar prior. order to ensure the fairness of the comparison, the pseudo-inputs size of VampPrior, the exemplars size of Exemplar prior and the pseudocoreset size of ByPE-VAF are set to the same value, which 500 in all of the experiments except 240 for CelebA. Since the data in CIFARIO is usually processed as continuous values, we preprocess all used datasets into continuous values the range of [0, in pursuit of uniformity. This is also beneficial for the pscudocoreset update. We further employ mean square (MSE) error as the reconstruction error. The results are shown in Table 1 that ByPE-VAES outperform other models in all cases.
from which one can
Method
Dynamic MNIST
Fashion MNIST
CIFARIO
VAE w/ Gaussian prior
24.41 + 0.06
21.43 + 0.10
72.21
VAE w/ VampPrior
23.65 + 0.03
20.87 + 0.01
71.97 0.05
VAE Exemplar prior
23.83 = 0.04
21.00
72.55 0.05
ByPE-VAK (ours)
23.61 + 0.03
20.85
71.91 0.02
HVAE w/ Gaussian prior
23.82 = 0.04
2.00
HVAE w/ VampPrior
23.54 + 0.03
0083+0000
71.54 0.04
HVAE w/ Exemplar prior
23.58 + 0.03
20.95 + 0.02
71.77 + 0.05
ByPE-HVAE (ours)
23.48+ 0.02
20.82+0.00
71.38+ 0.01
ConvH w/ Gaussian prior
23.16 0.05
2.00
70.83 +.83+0005
ConvH VAE w/ VampPrior
22.94 + 0.02
70.61 0.06
ConvHVAE w/ Exemplar prior
22.92 + 0.03
20.62 0.00
70.83 + 0.19
ByPE-ConvHVAE (ours)
22.84# 0.02
20.58+ 0.01
70.55+ 0.03
Table 1: Density estimation on Dynamic MNIST, Fashion MNIST, and CIFARIO based on different network architectures for four methods.
According to the generation process of ByPE-VAL VAE (as shown in Algorithm I. we generate a set of samples. given in Fig I The generated samples in each plate are based the same pseudodata point. As shown in Fig 1 SEPEEVVE can generate high-quality samples with various identifiable features of the data while inducing cluster without losing diversity. For the datasets with low diversity, such as MNIST and Fashion MNIST, these samples could retain the content of the data. For more diverse datasets (such as CelebA), although the details of the generated samples are different, they also show clustering effects on certain features, such as background, hairstyle, hair color, and face direction. This phenomenon is probably due the use of the pseudocoreset during the training phase. In addition, we conduct the interpolation in the latent space CelebA shown in Fig 2 which implies that the latent space learned by our model is smooth and meaningful.
5.2 Representation Learning
Since the structure of latent space also reveals the quality of the generative model, we further report the latent representation of BYPE-VAE. First, compare ByPE-VAE with VAE with Gaussian
7

---

5 5555
2 2
2 2 2 2 2 2 2
555 5
2 2 2
2 2
5555 5
5555
5555 5
2 2 2
2
5 5
2 2 2
2 2 2
5
5 5 5
2
(a) Dynamic MNIST
(b) Fashion MNIST
(c) CelebA
Figure I: Samples generated by ByPE-VAE based on the same pseudodata point in each plate. These show that ByPE-VAE can generate high-quality samples with various identifiable features of the data while inducing a cluster without losing diversity.
Figure Interpolation between samples from the CelebA dataset.
prior for the latent representations of MNIST test data. The corresponding t-SNE visualization is shown in F Test points with the same label are marked with the same color. For the latent representations of our model, the distance between classes is larger and the distance within classes is smaller. They are more meaningful than the representations of VAE. Then, we compare ByPE-VAE with the other three VAEs two datasets for the k-nearest neighbor (KNN) classification task. Fig shows the results for different values K, where K E (3,5, 5, 7, 9, 11, 13, 15). ByPE-VAE consistently outperforms other models MNIST and Fashion MNIST. Results on CIFAR10 reported on Supp since the space constraints.
(a) ByPE- MNIST (b) MNIST
(a) KNN on MNIST
(b) KNN on Fashion MNIST
Figure 3: t-SNE visualization of latent representations for test set, colored by labels.
Figure 4: kNN classification accuracy (%) with differ- ent values K on MNIST and Fashion MNIST.
5.3 Efficiency Analysis
We also compare ByPE-VAN with the Exemplar VAE from two aspects to further verify the efficiency of our model. First, we report the lig-likelihood values of two models for the test set, shown

---

ByPE NLL
VAE (500)
Vamp
VAE (500)
Exemplar VAE (25000)
Dataset
Time
NLL
Time
NLL
Time
Dynamic MNIST
23.61
13.19
23.65
13.03
23.61
35.45
Fashion MNIST
20.85
12.05
20.87
12.09
20.81
37.23
CIFARIO
71.91
17.30
71.97
14.89
72.00
66.85
Table 2: Resluts of average negative log-likelihood and training time (s/epoch) with various datasets.
in Fig. [] In the case of the same size of exemplars and pseudocoreset, the performance of ByPE- VAE is significantly better than Exemplar VAE. Second, we record the average training time of two models for each epoch. Here, M is set to 500, and the size of the exemplars is set to 25000, which is consistent with the value reported by Exemplar VAE. Note that in the part of the fixed pseudocoreset, the training time of SypPE-VAE is very small and basically the same as that of VAE with Gaussian prior. As result, the main time-consuming part of our model lies in the update of the pscudocoreset. However, the pseudocoreset needs to update every k epochs only while k is set to 10 our experiments. All experiments are run a single Nvidia 1080Ti GPU. The results can be seen in Table 2 performance.
where we find that our model obtains about 3x speed-up while almost holding the
5.4 Generative Data Augmentation
Finally, we evaluate the performance of ByPE-VAE for generating augmented data to further improve discriminative models. To be more comprehensive and fair, we adopt two ways to generate extra samples. The first way is to sample the latent representation from the posterior qs and then use it to generate a new sample for all models. The second way is to sample the latent representation from the prior P& and then use it to generate new sample. This way is only applicable to the ByPE-VAE and the Exemplar VAE. Note that, it is a little different from the generation process of our method in this task. Due the lack of labels in the pseudocoreset, we cannot directly let the prior P& be conditioned on the pscudocoreset. Specifically, we use the original training data to replace the pseudocoreset since the KI. divergence between Pz(z U, w) and po [ X) has become small at the end of the training. Additionally, the generated samples are labeled by corresponding original training data. We train the discriminative model mixture of original training data and generated data. Each training iteration is as follows, which refers to section 5.4 of [13.
Sample a minibatch X = {xx;, w:) ) == from training data.
For each x; E X, draw 2; ~ qo xx) or z; ~ ro(z xx), which correspond to two ways respectively.
For each 2,, set X; = pa 2.), which assigns the label ys, and then obtain a synthetic minibatch X = 8+)
Optimize the weighted cross entropy: {= E ( log po (yi xx) + (1 - X) log Pe (3t xx)].
As reported in 13), the hyper-parameter À is set 0.4. The network architecture of the discriminative model an MIP network with two hidden layers of 1024 units and ReL.U activations are adopted. The results summarized in Table 3 The test error of ByPE-VAE is lower than other models MNIST for both sampling ways.
Exemplar
BPE E-EA
Model
Test-error
Gaussian prior w/ Variational Posterior
1.23 + 0.02
Vampprior w/ Variational Posterior
1.20 + 0.02
Exemplar prior Variational Posterior
1.16 +
ByPE- VAF w/ Variational Posterior
1.10 + 0.01
Exemplar prior w/ Prior
25
ByPF- VAE w/ Prior
0.88+ 0.02
Average negative log- likelihood on test set with the different size of exemplars for Exemplar VAE and ByPE-VAL respectively.
Figure 5:
Table 3: Test error (%) on permutation invariant MNIST.

---

Conclusion
In this paper, we introduce ByPE- VAE, new variant of VAE with a Bayesian Pseudocoreset based prior. The proposed prior is conditioned on a small-scale meaningful pseudocoreset rather than large-scale training data, which greatly reduces the computational complexity and prevents overfitting. Additionally, through the variational inference formulation, we obtain the optimal pseudocoreset to approximate the entire dataset. For optimization, we employ a two-step alternative search strategy to optimize the parameters in the VAE framework and the pseudodata points along with weights in the pseudocoreset. Finally, we demonstrate the promising performance of the ByPE- VAE in a number of tasks and datasets.
Acknowledgment
This paper was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), and a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. [CYJ202001099 13403826). We thank Dr. Liangjian Wen for insightful discussion on the design of our frameworks.
References
[1] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[2] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastie backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286 PMLR, 2014.
[3] Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. arXiv preprint
2016.
[4] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement variational autoencoders. arXiv preprint 0rX1v:1880.049922.20 2018.
[5] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint arXiv:/511.06344 2015.
[6] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pages 1530-1538. PMLR, 2015.
[7] Jakub M Tomezak and Max Welling, Improving variational auto-encoders using householder flow. arXiv preprint arXiv:161 1.09630, 2016.
[8] Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving variational inference with inverse autoregressive flow. arXiv preprint 16Xiv:1606, 04934, 2016.
[9] Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: A latent variable model for natural images. arXiv preprint
2016.
[10] Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder, arXiv preprint 1rXiv:1611.0233) 2016.
[11] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume page 2016.
[12] Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223, PMLR, 2018
10

---

[13] Sajad Norouzi, David J Fleet, and Mohammad Norouzi. Exemplar VAE: Linking generative models, nearest neighbor retrieval, and data augmentation. Advances Neural Information Processing Systems, 33, 2020.
[14] Pankaj K Agarwal, Sariel Har-Peled, Kasturi R Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry, 52: 1-00, 2005.
[15] Dionysis Manousakas, Zuheng Xu, Cecilia Mascolo, and Trevor Campbell. Bayesian pseudo- coresets. Advances in Neural Information Processing Systems, 2020.
[16] Alireza Makhzami, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adver- sarial autoencoders. arXiv preprint
2015.
[17] Lirong He, Bin Liu, Guangxi Li, Yongpan Sheng, Yafang Wang, and Zenglin Xu. Knowledge base completion by variational bayesian neural tensor decomposition. Cognitive Computation, 10(6):0755-1084 2018.
[18] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 66-75. PMLR, 2019.
[19] Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. arXiv preprint 16Xiv: 2016.
[20] Prasoon Goyal, Zhiting Hu, Xiaodan Liang, Chenyu Wang, and Eric Xing. Nonparametric variational auto-encoders for hierarchical representation learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 5094 5102, 2017.
[21] Jörg Bornschein, Andriy Mnih, Daniel Zoran, and Danilo Rezende. Variational memory addressing in generative models. arXiv preprint arXiv: 16, 2017.
[22] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autocncoders. arXiv preprint arXiv:1903. 2019.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6980, 2014.
[24] Adams Wei Yu, Lei Huang, Qihang Lin, Ruslan Salakhutdinov, and Jaime Carbonell. Block- normalized gradient method: An empirical study for training deep neural network. arXiv preprint 1rXiv:1707.000 04802, 2017.
[25] Adams Wei Yu, Qihang Lin, Ruslan Salakhutdinov, and Jaime Carbonell. Normalized gradient with adaptive stepsize method for deep neural network training. arXiv preprint arXiv: 048222 1(1), 2017.
[26] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings. 2010.
(27] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv: 150900050900099, 2015.


Reviews:
[{"review_id": "wtTiiAoSEkk", "reviewer": "Reviewer_LzjC", "summary": "This paper proposes the use of pseudo-coresets to accelerate inference in Examplar VAE.\nThe Examplar VAE ELBO (Eq.4/5 in [ExamplarVAE]) is updated to use pseudo-coreset (Eq.8/9 in this paper). Other quantities of interest are updated accordingly and a two-stage approach is proposed to update two types of parameters (VAE and pseudo-coresets).\n\n[ExamplarVAE] Norouzi et al (2020) \"Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation\", NeurIPS 2020.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors do not report the sensitivity for the parameter 'k' (pseudocoreset are updated every 'k' epochs).\nIt would be interesting to show a few different values to understand how important these updates are, for example how data-dependent. \nIn KNN the number of neighboors is also 'K' which can be confusing.\n\nThe dynamics of the two-stage optimization approach are not examined. This would be important as it is a key novel component of the proposed method. Is convergence stable across experimental settings? does it requires any tuning?\n\nFig.5: are the differences in log-likelihood statistically significant?\n\nOverall, the goal is clear and the methodology is rigorously detailed, including all equations of interests. \nMy main concern is novelty: the method is a relatively straightforward extension of Examplar VAE using Pseudo-coresets - both of these techniques are pre-existent.\nAdditionally, it would have been good to demonstrate this approach on a larger/more challenging dataset.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "r56qF-MVp3E", "reviewer": "Reviewer_X1mj", "summary": "The paper tackles the important problem of finding suitable learnable prior for VAE. The paper provides some improvements of Exemplar VAE that increase the efficiency of the model. The proposed model is simply aiming at reducing the examples for nearest neighbours used by Exemplar VAE by application of bayesian pseudocoresets  for efficiently approximating the entire original dataset. The proposed approach was evaluated using standard benchmark datasets in a number of tasks, including density estimation, representation learning and generative data augmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The main contribution of the paper is well-expressed and seems to be well-motivated. I like the application for data augmentation - however it requires more experimental analysis. \n\nThe proposed approach seems to be a bit incremental compared to Exemplar VAE  and VAMPrior. Conceptually, the paper presents the alternative way (referring to VamPrior) of learning pseudo inputs for the Exemplar VAE model. It would be beneficial to make some deeper comparison (both theoretical nad empirical) between the nature of pseudo inputs from VAMPrior and ByPE-VAE and express the novelty. \n\nAfter reading the introduction and method description I would expect more convincing results of experiments. It is not clear, what type of criterion is provided in table 1. It seems that it is test NLL estimated using the Importance Sampling. Why pure ELBO value is not used for evolution? For CelebA there are no quantitative results reported in Table 1. but this dataset was used in qualitative analysis. \n\nFor the remaining experiments the range of datasets is reduced to MNIST and related. Having the models trained on CIFAR10 and CelebA it would be beneficial to see complete results for augmentation and representation tasks, even some of them are not SOTA compared to reference approaches. Similar issue is observed with table 2. - why results for VAMP (assuming the same number of pseudo inputs) are not reported here. \n\nBecause I serve as emergency reviewer I did not have to chance to deeply go through the math formulas but they seems to be correct. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not notice the limitations of proposed approach - it should be expressed in more visible way. The negative aspects are not discussed by authors - also needs including. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "FgfjoWOacEH", "reviewer": "Reviewer_rLyr", "summary": "The paper proposes a Bayesian pseudocoreset based exemplar VAE algorithm which utilizes pseudocoresets to improve the computational complexity and reduce over-fitting of Exemplar-VAE (a related work), by conditioning the prior on a subset of data. The paper proposes variational inference to find the optimal pseudocoreset, by minimizing the KL divergence between the posteriors defined on the pseudocoreset and the training data. Detailed experimental evaluation is provided against baselines and competitors and shows improved performance of the proposed approach in running time and quality of the learned latent space.\n\nThe main contributions of the paper are:\n- Bayesian pseudocoresets proposed instead of using all the training data as exemplars in  Exemplar-VAE\n- Simultaneous optimization of the optimal pseudocoreset and VAE training, with the optimal pseudocoreset selected through variational inference\n- Detailed experimental evaluation against competitors showing improved running time and latent space quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes Bayesian pseudocoresets to improve the computational complexity of Exemplar-VAE and to reduce overfitting.  The paper is well-written, and extensive experimental evaluation is provided. Variational inference to find optimal pseudocoresets, and the joint optimization along with VAE are strong technical contributions. However, the paper very closely follows the structure, formatting, and experimental evaluation of Exemplar-VAE (a related work). New experimental evaluation or insights on the pseudocoresets are not provided but a repetition of the same experiments conducted by Exemplar-VAE on the proposed approach. The novelty of the paper is thus somewhat limited and incremental. The pseudocode for the algorithm optimization is not explained in the text.\n\nDetailed comments:\n\n- The optimization process in Algorithm 2 is not explained in the paper. What is the significance of subsets S and M in Algorithm 2?\n- Equations (9)-(14) are similar or same as the ones in Exemplar-VAE. The equations defined in Exemplar-VAE should be cited.  \n- Statistical significance of results on KNN is not provided. The results on density estimation show marginal improvement for the proposed approach.\n- The t-SNE plots should also compare the latent space of Exemplar-VAE. Currently only the proposed approach and a baseline VAE are compared.\n- Experiments to show how well the learned pseudocoresets approximate the training data, would be a useful addition.\n- A comparison of KNN/log-likelihood for different coreset sizes would be useful. How does the proposed approach perform if the coreset size is set equal to the exemplar size in Exemplar-VAE?\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed, no negative societal impact observed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new way of learning a prior in the Variational Auto-Encoder framework. The paper is a follow-up on the VampPrior VAE and the Exemplar VAE. In a nutshell, the idea is to use a Bayesian pseudocoreset to learn pseudoinputs (in the context of the VampPrior). Overall, the reviewers appreciate the paper because:
- It is easy to follow.
- All concepts are clearly explained.
- The idea of using the Byesian pseudocoreset is interesting.

The main concern is about the experimental section. The reviewers would prefer to see a deepr comparison of learned pseudoinputs using backpropagation (like in the VampPrior) or by the proposed approach, and how learned pseudoinputs compared to real data.
The rebuttal contains many new results and addresses many concerns raised by the reviewers. Therefore, I tend to accept the paper.

Decision: Accept (Poster)