ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
Year: 2022

Abstract:
Recent studies show that advanced priors play a major role in deep generative models. Exemplar VAE, as a variant of VAE with an exemplar-based prior, has achieved impressive results. However, due to the nature of model design, an exemplar-based model usually requires vast amounts of data to participate in training, which leads to huge computational complexity. To address this issue, we propose Bayesian Pseudocoresets Exemplar VAE (ByPE-VAE), a new variant of VAE with a prior based on Bayesian pseudocoreset. The proposed prior is conditioned on a small-scale pseudocoreset rather than the whole dataset for reducing the computational cost and avoiding overfitting. Simultaneously, we obtain the optimal pseudocoreset via a stochastic optimization algorithm during VAE training aiming to minimize the Kullback-Leibler divergence between the prior based on the pseudocoreset and that based on the whole dataset. Experimental results show that ByPE-VAE can achieve competitive improvements over the state-of-the-art VAEs in the tasks of density estimation, representation learning, and generative data augmentation. Particularly, on a basic VAE architecture, ByPE-VAE is up to 3 times faster than Exemplar VAE while almost holding the performance. Code is available at https://github.com/Aiqz/ByPE-VAE.

heading: 1 Introduction
text: Deep generative models that learn implicit data distribution from the enormous amount of data have received widespread attention to generating highly realistic new samples in machine learning. In particular, due to the utilization of the reparameterization trick and variational inference for optimization, Variational Autoencoders (VAEs) [1, 2] stand out and have demonstrated significant successes for dimension reduction [3], learning representations [4], and generating data [5]. In addition, various variants of VAE have been proposed conditioned on advanced variational posterior [6, 7, 8] or powerful decoders [9, 10].
It is worth noting that the prior in the typical VAE is a simple standard normal distribution that is convenient to compute while ignores the nature of the data itself. Moreover, a large number of experiments have empirically demonstrated that simplistic priors could produce the phenomena of over-regularization and posterior collapse, and finally cause poor performance [5, 11]. Hence, many researchers have worked to develop more complex priors to enhance the capacity of the variational posterior. In this line, Tomczak et al. [12] introduces a more flexible prior named VampPrior, which is a mixture of variational posteriors based on pseudo-inputs to alleviate the problems like overfitting
∗Corresponding Author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
and high computational cost. However, the way in which the pseudo-inputs are obtained is not interpretable. Recently, Norouzi et al. [13] develops an Exemplar VAE with a non-parametric prior based on an exemplar-based method, achieving excellent performance. To ensure the performance and the generation diversity, the exemplar set needs to be large enough, and usually the entire training data is utilized. Obviously, this leads to huge computational complexity. Even though Exemplar VAE further employs approximate nearest neighbor search to speed up the training process, the number of nearest neighbors should be as large as possible to ensure performance. In a nutshell, Exemplar VAE is computationally expensive.
To address such issues, we develop a new prior for VAE that is inspired by the paradigm of coresets [14]. The coreset is a powerful tool aiming to find a small weighted subset for efficiently approximating the entire original dataset. Therefore, rather than using the large-scale training data directly, we seek to design a prior conditioned on a coreset, which greatly reduces the computational complexity and prevents overfitting. In practice, to better incorporate this idea with the framework of VAE, we further employ a specific form of coresets, namely Bayesian pseudocoresets [15], which is known as a small weighted subset of the pseudodata points, resulting in a Bayesian pseudocoreset based prior. With this prior, we gain a new variant of VAE called Bayesian Pseudocoresets Exemplar VAE (ByPE-VAE). To sample from the ByPE-VAE, we first take a pseudodata point from the pseudocoreset according to its weight, and then transform it into a latent representation using the learned prior. Then a decoder is used to transform the latent representation into a new sample.
A crucial part of ByPE-VAE is to obtain the optimal pseudocoreset. We formulate this process as a variational inference problem where the pseudodata points and corresponding weights are the parameters of variational posterior approximation. More precisely, to seek the optimal pseudocoreset, we minimize the Kullback-Leibler (KL) divergence between the prior based on the pseudocoreset and that based on the entire dataset. This processing ensures that the learned prior is actually an approximation of the prior conditioned on whole training data. Thus, it is fundamentally different from general pseudodata based priors, like VampPrior. For optimization, we adopt a two-step alternative search strategy to learn two types of parameters of ByPE-VAE, which refer to the parameters in the VAE framework and the pseudodata points and corresponding weights in the pseudocoreset. In particular, we iteratively optimize one of the two types of parameters while keeping the other one fixed until convergence.
Finally, we compare ByPE-VAE with several state-of-the-art VAEs in a number of tasks, including density estimation, representation learning and generative data augmentation. Experimental results demonstrate the effectiveness of ByPE-VAE on Dynamic MNIST, Fashion MNIST, CIFAR10, and CelebA. Additionally, to validate the efficiency of our model, we measure the running time on a basic VAE architecture. Compared to the Exemplar VAE, ByPE-VAE is up to 3 times speed-up without losing performance on Dynamic MNIST, Fashion MNIST, and CIFAR10.

heading: 2 Preliminaries
text: Before presenting the proposed model, we first introduce some preliminaries, namely Exemplar VAE and Bayesian pseudocoresets. Throughout this paper, vectors are denoted by bold lowercase letters, whose subscripts indicate their order, and matrices are denoted by upper-case letters.

heading: 2.1 Exemplar VAE
text: Exemplar VAE is regarded as a variant of VAE that integrates the exemplar-based method into VAE for the sake of seeking impressive image generation. Specifically, it first draws a random exemplar xn using uniform distribution from the training data X = {xn}Nn=1 (where N denotes the sample amount), then transforms an exemplar xn into latent variable z using an example-based prior rφ(z | xn), finally generates observable data x by a decoder pφ(x | z). The parametric transition distribution Tφ,θ(x | xn) of exemplar-based generative models can be expressed as
Tφ,θ(x | xn) = ∫ z rφ(z | xn)pθ(x | z)dz, (1)
where φ and θ denote corresponding parameters. Assuming that x is independent to xn conditioned on the latent variable z and marginalizing over the latent variable z, the objective O(θ, φ;x, X) of
Exemplar VAE can be formulated as
log p(x;X, θ, φ) = log N∑ n=1 1 N Tφ,θ(x|xn) ≥ Eqφ(z|x) log pθ(x|z)− Eqφ(z|x) log qφ(z|x)∑N n=1 rφ(z|xn)/N
= O(θ, φ;x, X), (2) where O(θ, φ;x, X) is known as the evidence lower bound (ELBO). From the Eq. (2), it can be derived that the difference between Exemplar VAE and typical VAE is the definition of the prior p(z) in the second term. The prior of Exemplar VAE is defined as a mixture form, that is p(z | X) = ∑N n=1 rφ(z | xn)/N . The variational posterior qφ(z | x) and the exemplar-based prior rφ(z | xn) are assumed to be Gaussian distributions whose parameters are fulfilled by neural networks. Note that, the computational cost of this training process is related to the number of exemplars which usually set to be the entire training data. This indicates that Exemplar VAE is computationally expensive since the amount of training data is generally huge.

heading: 2.2 Bayesian Pseudocoresets
text: Bayesian pseudocoresets is a method of coreset construction based on variational inference, which constructs a weighted set of synthetic “pseudodata” instead of the original dataset during inference. First, the goal of this method is to approximate expectations under the posterior π(ψ) with the parameter ψ, which is formulated as
π(ψ) = 1
Z exp ( N∑ n=1 f(xn, ψ) ) π0(ψ), (3)
where f(·, ψ) denotes a potential function, and usually is a log-likelihood function. π0(ψ) is the prior, and Z is the normalization constant. Instead of directly inferring the posterior π(ψ), the Bayesian pseudocoreset employs a weighted set of pseudodata points to approximate the true posterior π(ψ), which is given by
πU,w(ψ) = 1
Z̃U,w exp ( M∑ m=1 wmf(um, ψ) ) π0(ψ), (4)
where U = {um}Mm=1 represents M pseudodata points um ∈ Rd, w = {wm}Mm=1 denotes nonnegative weights, and Z̃U,w is the corresponding normalization constant. Finally, this model obtains the optimal pseudodata points and their weights by minimizing the KL divergence, as follows,
U?,w? = argmin U,w DKL (πU,w‖π) . (5)
This formulation can reduce the computational cost by decreasing data redundancy.

heading: 3 Bayesian Pseudocoresets Exemplar VAE
text: To generate a new observation x, the Exemplar VAE requires a large collection of exemplars from X = {xn}Nn=1 to guide the whole process, as shown in Eq. (2). One can see that the greater the number of exemplars set, the richer the prior information can be obtained. In practice, to ensure performance, the number of exemplars is relatively large which is generally set to the size of the entire training data. This leads to huge computational costs in the training process of the Exemplar VAE. To overcome such a issue, inspired by the paradigm of Bayesian pseudocoresets, we adopt M pseudodata points U = {um}Mm=1 with corresponding weights w = {wm}Mm=1 to denote exemplars, importantly M N . That is, the original exemplars X are approximated by a small weighted set of pseudodata points known as a Bayesian pseudocoreset. The framework can be expressed as
log p(x | X, θ) = log N∑ n=1 1 N Tθ(x | xn) ≈ log M∑ m=1 wm N Tθ(x | um) = log p(x | U,w, θ), (6)
wherewm ≥ 0 (m = 1, · · · ,M) and ‖w‖1 = N . Further, we integrate this approximated framework with VAE by introducing a latent variable z. Then the parametric function is given by
Tφ,θ(x | um) = ∫ z rφ(z | um)pθ(x | z)dz, (7)
where rφ(z | um) with parameter φ denotes a pseudodata based prior for generating z from a pseudodata point um. pθ(x | z) with parameter θ represents the decoder for generating the observation x from z. Similarly, we assume that an observation x is independent from a pseudodata point um conditional on z to simplify the formulation and optimization.
In general, we desire to maximize the marginal log-likelihood log p(x) for learning, however, this is intractable since we have no ability to integrate the complex posterior distributions out. Now, we focus on maximizing the evidence lower bound (ELBO) derived by Jensen’s inequality, as follows,
log p(x;U,w, θ, φ) = log M∑ m=1 wm N Tφ,θ(x | um) = log M∑ m=1 wm N ∫ z rφ(z | um)pθ(x | z)dz (8)
≥ Eqφ(z|x) log pθ(x | z)− Eqφ(z|x) log qφ(z | x)∑M m=1 wmrφ(z | um)/N ≡ O(θ, φ, U,w;x), (9)
where qφ(z | x) represents the approximate posterior distribution. And O(θ, φ, U,w;x) is defined as the objective function of the ByPE-VAE to optimize parameters θ and φ. The specific derivation can be found in Supp.A. As we can see from Eq. (9), the difference between the ByPE-VAE and other variants of VAE is the formulation of the prior p(z) in the second term. In detail, the prior of the ByPE-VAE is a weighted mixture model prior, in which each component is conditioned on a pseudodata point and the corresponding weight, i.e., p(z|U,w) = ∑M m=1 wmrφ(z|um)/N . In
contrast, the prior of the Exemplar VAE is denoted as p(z|X) = ∑N n=1 rφ(z|xn)/N .
As shown in Eq. (9), the ByPE-VAE includes two encoder networks, namely qφ(z | x) and rφ(z | um). And the distributions of qφ(z | x) and rφ(z | um) are both designed as Gaussian distributions. According to the analysis in [16], the optimal prior is the form of aggregate posterior. Inspired by this report, we make the prior be coupled with the variational posterior, as follows,
qφ(z | x) = N (z | µφ(x),Λφ(x)), (10) rφ(z | um) = N (z | µφ(um), σ2I). (11)
We employ the same parametric mean function µφ for two encoder networks for better incorporation. However, the covariance functions of the two encoder networks are different. Specifically, the variational posterior uses a diagonal covariance matrix function Λφ, while each component of the mixture model prior uses an isotropic Gaussian with a scalar parameter σ2. Note that the scalar parameter σ2 is shared by each component of the mixture model prior for effective computation. Then, we can express the log of the weighted pseudocoreset based prior log pφ(z | U,w) as
log pφ(z | U,w) = −dz log( √ 2πσ)− logN + log M∑ m=1 wm exp −‖z− µφ (um)‖2 2σ2 , (12)
where dz denotes the dimension of z. Based on the formulation of Eq. (12), we can further obtain the objective function of the ByPE-VAE, as follows,
O(θ, φ, U,w;X) = Eqφ(z|x)
[ log
pθ(x | z) qφ(z | x) + log M∑ m=1
wm
( √ 2πσ)dz exp −‖z− µφ (um)‖2 2σ2
] ,
(13)
∝ N∑ i=1 Eqφ(z|xi)
[ log
pθ(xi | z) qφ(z | xi) + log M∑ m=1
wm
( √ 2πσ)dz exp −‖z− µφ (um)‖2 2σ2
] ,
(14)
where the constant − logN is omitted for convenience. For Eqφ(z|xi), we employ the reparametrization trick to generate samples. Note that, for the standard VAE with a Gaussian prior, the process of generating new observations involves only the decoder network after training. However, to generate a new observation from ByPE-VAE, we not only require the decoder network, but also the learned Bayesian pseudocoreset and a pseudodata point based prior rφ. We summarize the generative process of the ByPE-VAE in Algorithm 1.
Algorithm 1: The Generative Process of ByPE-VAE Input : Pseudocoreset {U,w}, Decoder pθ , Prior rφ. Output :A generated observation x. Step.1 Sample um ∼ Multi(· | U, wN ) for obtaining a pseudodata point um from the learned pseudocoreset. Step.2 Sample z ∼ rφ(· | um) using the pseudodata point based prior rφ for obtaining the latent
representation z. Step.3 Sample x ∼ pθ(· | z) using the decoder pθ for generating a new observation x.
However, more importantly, the whole process above holds only if the pseudocoreset is an approximation of all exemplars or training data. To ensure this, we first re-represent frameworks of pφ(z | X) and pφ(z | U,w) from the Bayesian perspective. Concretely, pφ(z | X) and pφ(z | U,w) are also viewed as posteriors conditioned on the likelihood function pθ(· | z) and a certain prior p0(z), as follows,
pφ(z | X) = 1
Z exp ( N∑ n=1 log pθ(xn | z) ) p0(z), (15)
pφ(z | U,w) = 1
Z̃U,w exp ( M∑ m=1 wm log pθ(um | z) ) p0(z), (16)
where Z and Z̃U,w are their respective normalization constants, pθ(· | z) here specifically refers to the decoder. Then, we develop this problem into a variational inference problem, where the pseudodata points and the corresponding weights are the parameters of the variational posterior approximation followed [15]. Specifically, we construct the pseudocoreset by minimizing the KL divergence in terms of the pseudodata points and the weights, as follows,
U?,w? = argmin U,w DKL (pφ(z | U,w)‖pφ(z | X)) . (17)
Further, the gradients of DKL in Eq. (17) with respect to the pseudodata point um and the weights w are given by
∇umDKL = −wm CovU,w [ ∇U log pθ(um | z), log pθ(X | z)T1N − log pθ(U | z)Tw ] , (18)
∇wDKL = −CovU,w [ log pθ(U | z), log pθ(X | z)T1N − log pθ(U | z)Tw ] , (19)
where CovU,w denotes the covariance operator for the pφ(z | U,w), and 1N ∈ RN represents the vector of all 1 entries. In practice, we adopt a black-box stochastic algorithm to obtain the optimal pseudodata points and weights. Details of these derivations are provided in Supp.B and C.
Note that ByPE-VAE involves two types of parameters, namely the parameters θ and φ in the VAE framework and the parameters U and w in the pseudocoreset. For optimization, we use a two-step alternative optimization strategy. In detail, (i) update θ and φ with fixed pseudocoreset {U,w}, and (ii) update U and w with fixed θ and φ. Steps (i) and (ii) are iteratively implemented until convergence. The detailed optimization algorithm is shown in Algorithm 2. One can see that the computation does not scale with N , but rather with the number of pseudocoreset points M , which greatly reduces the computational complexity and also prevents overfitting. Also note that, rather than be updated every epoch, the pseudocoreset {U,w} is updated by every k epochs. And k is set to 10 in the experiments.

heading: 4 Related Works
text: Variational Autoencoders (VAEs) [1, 2] are effectively deep generative models that utilize the variational inference and reparameterization trick for dimension reduction [3], learning representations [4], knowledge base completion[17], and generating data [5]. And various variants of VAE have been proposed conditioned on advanced variational posterior [6, 7, 8], powerful decoders [9, 10] and flexible priors [10, 18, 12, 13]. As for the prior, the standard VAE takes the normal distribution as the prior, which may lead to the phenomena of over-regularization and posterior collapse and further affect the performance for density estimation [5, 11]. In the early stage, VAEs apply more complex
Algorithm 2: The Optimization Algorithm for ByPE-VAE Input :Training data X ≡ {xn}Nn=1, batch size B, training epochs T , learning rate γt, sample size S,
pseudocoreset size M , update interval k Decoder pθ , variational posterior qφ, weighted pseudocoreset based prior pφ Initialized pseudocoreset by which B ∼ UnifSubset ([N ],M),B := {b1, . . . , bM} um ← xbm , wm ← N/M, m = 1, · · · ,M
Output :Parameters θ and φ, Pseudocoreset {U,w} 1 for t = 1, · · · , T do /* Optimize VAE parameters θ and φ */ 2 w← w + (N/M −w.mean) Centralized to w 3 Evaluate the ByPE-VAE objective using Eq. (14), and update θ and φ using the ADAM /* Optimize pseudocoreset um and wm,m = 1, · · · ,M */ 4 if t/k = 0 then 5 Take S samples from current pseudocoreset posterior pφ(z|U,w), namely (z)Ss=1 ∼ pφ(z|U,w) 6 Obtain a mini-batch of B datapoints B ∼ UnifSubset ([N ], B) 7 for s = 1, ..., S do 8 gs ← ( log pθ(xb|zs)− 1/S ∑S s′=1 log pθ(xb|zs′) ) b∈B ∈ RB
9 g̃s ← ( log pθ(um|zs)− 1/S ∑S s′=1 log pθ(um|zs′) )M m=1 ∈ RM
10 for m = 1, ...,M do 11 h̃m,s ← ∇U log pθ(um|zs)− 1/S ∑S s′=1∇U log pθ(um|zs′)) ∈ R d
12 ∇̂w ← −1/s ∑S s=1 g̃s ( N/BgTs 1− g̃Ts w ) 13 for m = 1, ...,M do 14 ∇̂um ← −w1m/S ∑S s=1 h̃m,s ( N/BgTs 1− g̃Ts w
) 15 w← max ( w − γt∇̂w, 0
) 16 for m = 1, ...,M do 17 um ← um − γt∇̂um
priors, such as the Dirichlet process prior [19], the Chinese Restaurant Process prior [20], to improve the capacity of the variational posterior. However, these methods can only be trained with specific tricks and learning methods. Chen et al. [10] employs the autoregressive prior which is then along with a convolutional encoder and an autoregressive decoder to ensure the performance of generation.
Tomczak et al. [12] introduces a variational mixture of posteriors prior (VampPrior) conditioned on a set of pseudo-inputs aiming at approximating the aggregated posterior. The intent of our method is similar to the VampPrior with respect to the use of pseudo-inputs. Nevertheless, there is a fundamental difference. The pseudo-inputs in [12] are regarded as hyperparameters of the prior and are obtained through backpropagation, while the pseudo-inputs of our model are the Bayesian pseudocoreset and are optimized through variational inference. Therefore, the pseudo-inputs learned by our model could approximate all the original data, with the weighting operation carried on. In other words, our model is easier to understand in terms of interpretability.
Exemplar VAE [13] is a variant of VAE with a non-parametric prior based on an exemplar-based method to learn desirable hidden representations. Exemplar VAE takes all training data to its exemplar set instead of pseudo-inputs. This computational cost is expensive since the amount of training data can be huge. Hence, Exemplar VAE further presents the approximate kNN search to reduce the cost. However, this technique could reduce the effectiveness of the algorithm and is used only in the training process. Our model introduces the Bayesian pseudocoreset under the Exemplar VAE framework, which improves not only the computational speed, but also the performance of VAEs.
In addition, a memory-augmented generative model with the discrete latent variable [21] is proposed to improve generative models. Our model can be considered as a VAE with additional memory. There are two essential differences between our model and [21]. First, the pseudo-inputs are based on the Bayesian pseudocoreset which is easy to interpret. Second, our model doesn’t need a normalized categorical distribution.

heading: 5 Experiments
text: Experimental setup We evaluate the ByPE-VAE on four datasets across several tasks based on multiple network architectures. Specifically, the tasks involve density estimation, representation learning and data augmentation, the used four datasets include MNIST, Fashion-MNIST, CIFAR10, and CelebA, respectively. Following [13], for the first three datasets, we conduct experiments on three different VAE architectures, namely a VAE based on MLP with two hidden layers, an HVAE based on MLP with two stochastic layers, and a ConvHVAE based on CNN with two stochastic layers. Following [22], we adopt the convolutional architecture for CelebA. In addition, we measure the running time on the first network architecture for three datasets. The ADAM algorithm with normalized gradients [23, 24, 25] is used for optimization and learning rate is set to 5e-4. And we use KL annealing for 100 epochs and early-stopping with a look ahead of 50 epochs. In addition, the weights of the neural networks are initialized according to [26]. Following [12] and [13], we use Importance Weighted Autoencoders (IWAE) [27] with 5000 samples for density estimation.

heading: 5.1 Density Estimation
text: To validate the effectiveness of ByPE-VAE, we compare ByPE-VAE with state-of-the-art methods for each architecture, namely a Gaussian prior, a VampPrior, and an Exemplar prior. In order to ensure the fairness of the comparison, the pseudo-inputs size of VampPrior, the exemplars size of Exemplar prior and the pseudocoreset size of ByPE-VAE are set to the same value, which is 500 in all of the experiments except 240 for CelebA. Since the data in CIFAR10 is usually processed as continuous values, we preprocess all used datasets into continuous values in the range of [0, 1] in pursuit of uniformity. This is also beneficial for the pseudocoreset update. We further employ mean square (MSE) error as the reconstruction error. The results are shown in Table 1, from which one can see that ByPE-VAEs outperform other models in all cases.
According to the generation process of ByPE-VAE (as shown in Algorithm 1), we generate a set of samples, given in Fig.1. The generated samples in each plate are based on the same pseudodata point. As shown in Fig.1, ByPE-VAE can generate high-quality samples with various identifiable features of the data while inducing a cluster without losing diversity. For the datasets with low diversity, such as MNIST and Fashion MNIST, these samples could retain the content of the data. For more diverse datasets (such as CelebA), although the details of the generated samples are different, they also show clustering effects on certain features, such as background, hairstyle, hair color, and face direction. This phenomenon is probably due to the use of the pseudocoreset during the training phase. In addition, we conduct the interpolation in the latent space on CelebA shown in Fig.2, which implies that the latent space learned by our model is smooth and meaningful.

heading: 5.2 Representation Learning
text: Since the structure of latent space also reveals the quality of the generative model, we further report the latent representation of ByPE-VAE. First, we compare ByPE-VAE with VAE with a Gaussian
(a) Dynamic MNIST (b) Fashion MNIST
prior for the latent representations of MNIST test data. The corresponding t-SNE visualization is shown in Fig.3. Test points with the same label are marked with the same color. For the latent representations of our model, the distance between classes is larger and the distance within classes is smaller. They are more meaningful than the representations of VAE. Then, we compare ByPE-VAE with the other three VAEs on two datasets for the k-nearest neighbor (kNN) classification task. Fig.4 shows the results for different values of K, where K ∈ {3, 5, 7, 9, 11, 13, 15}. ByPE-VAE consistently outperforms other models on MNIST and Fashion MNIST. Results on CIFAR10 are reported on Supp.F since the space constraints.

heading: 5.3 Efficiency Analysis
text: We also compare ByPE-VAE with the Exemplar VAE from two aspects to further verify the efficiency of our model. First, we report the log-likelihood values of two models for the test set, as shown
in Fig. 5. In the case of the same size of exemplars and pseudocoreset, the performance of ByPEVAE is significantly better than Exemplar VAE. Second, we record the average training time of two models for each epoch. Here, M is set to 500, and the size of the exemplars is set to 25000, which is consistent with the value reported by Exemplar VAE. Note that in the part of the fixed pseudocoreset, the training time of ByPE-VAE is very small and basically the same as that of VAE with a Gaussian prior. As a result, the main time-consuming part of our model lies in the update of the pseudocoreset. However, the pseudocoreset needs to update every k epochs only while k is set to 10 in our experiments. All experiments are run on a single Nvidia 1080Ti GPU. The results can be seen in Table 2, where we find that our model obtains about 3× speed-up while almost holding the performance.

heading: 5.4 Generative Data Augmentation
text: Finally, we evaluate the performance of ByPE-VAE for generating augmented data to further improve discriminative models. To be more comprehensive and fair, we adopt two ways to generate extra samples. The first way is to sample the latent representation from the posterior qφ and then use it to generate a new sample for all models. The second way is to sample the latent representation from the prior pφ and then use it to generate a new sample. This way is only applicable to the ByPE-VAE and the Exemplar VAE. Note that, it is a little different from the generation process of our method in this task. Due to the lack of labels in the pseudocoreset, we cannot directly let the prior pφ be conditioned on the pseudocoreset. Specifically, we use the original training data to replace the pseudocoreset since the KL divergence between pφ(z | U,w) and pφ(z | X) has become small at the end of the training. Additionally, the generated samples are labeled by corresponding original training data. We train the discriminative model on a mixture of original training data and generated data. Each training iteration is as follows, which refers to section 5.4 of [13],
• Sample a minibatch X = {(xi, yi)}Bi=1 from training data. • For each xi ∈ X , draw zi ∼ qφ(z | xi) or zi ∼ rφ(z | xi), which correspond to two ways
respectively. • For each zi, set x̃i = pθ(x | zi), which assigns the label yi, and then obtain a synthetic minibatch X̃ = {(x̃i, yi)}Bi=1. • Optimize the weighted cross entropy: ` = − ∑B i=1 [λ log pθ (yi | xi) + (1− λ) log pθ (yi | x̃i)].
As reported in [13], the hyper-parameter λ is set to 0.4. The network architecture of the discriminative model is an MLP network with two hidden layers of 1024 units and ReLU activations are adopted. The results are summarized in Table 3. The test error of ByPE-VAE is lower than other models on MNIST for both sampling ways.

heading: 6 Conclusion
text: In this paper, we introduce ByPE-VAE, a new variant of VAE with a Bayesian Pseudocoreset based prior. The proposed prior is conditioned on a small-scale meaningful pseudocoreset rather than large-scale training data, which greatly reduces the computational complexity and prevents overfitting. Additionally, through the variational inference formulation, we obtain the optimal pseudocoreset to approximate the entire dataset. For optimization, we employ a two-step alternative search strategy to optimize the parameters in the VAE framework and the pseudodata points along with weights in the pseudocoreset. Finally, we demonstrate the promising performance of the ByPE-VAE in a number of tasks and datasets.

heading: Acknowledgment
text: This paper was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), and a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826). We thank Dr. Liangjian Wen for an insightful discussion on the design of our frameworks.

Review summaries:
The paper proposes a Bayesian pseudocoreset based exemplar VAE algorithm which utilizes pseudocoresets to improve the computational complexity and reduce over-fitting of Exemplar-VAE (a related work), by conditioning the prior on a subset of data. The paper proposes variational inference to find the optimal pseudocoreset, by minimizing the KL divergence between the posteriors defined on the pseudocoreset and the training data. Detailed experimental evaluation is provided against baselines and competitors and shows improved performance of the proposed approach in running time and quality of the learned latent space.  The main contributions of the paper are: - Bayesian pseudocoresets proposed instead of using all the training data as exemplars in  Exemplar-VAE - Simultaneous optimization of the optimal pseudocoreset and VAE training, with the optimal pseudocoreset selected through variational inference - Detailed experimental evaluation against competitors showing improved running time and latent space quality.

The paper tackles the important problem of finding suitable learnable prior for VAE. The paper provides some improvements of Exemplar VAE that increase the efficiency of the model. The proposed model is simply aiming at reducing the examples for nearest neighbours used by Exemplar VAE by application of bayesian pseudocoresets  for efficiently approximating the entire original dataset. The proposed approach was evaluated using standard benchmark datasets in a number of tasks, including density estimation, representation learning and generative data augmentation.

This paper proposes the use of pseudo-coresets to accelerate inference in Examplar VAE. The Examplar VAE ELBO (Eq.4/5 in [ExamplarVAE]) is updated to use pseudo-coreset (Eq.8/9 in this paper). Other quantities of interest are updated accordingly and a two-stage approach is proposed to update two types of parameters (VAE and pseudo-coresets).  [ExamplarVAE] Norouzi et al (2020) "Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation", NeurIPS 2020.