Hierarchical Bayes Autoencoders
Authors: Shuangfei Zhai
Carlos Guestrin
Joshua M. Susskind
Venue:  (2020)

Abstract: Autoencoders are powerful generative models for complex data, such as images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible semantic variations in the space of images. To address this problem, we present a new probabilistic generative model called the \emph{Hierarchical Bayes Autoencoder (HBAE)}. The HBAE contains a multimodal decoder in the form of an energy-based model (EBM), instead of the commonly adopted unimodal Gaussian distribution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where a conditional generator is trained to match the EBM distribution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the conditional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for a set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and generates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able to generate complex image sets.

Under review as a conference paper at ICLR 2020
HIERARCHICAL BAYES AUTOENCODERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Autoencoders are powerful generative models for complex data, such images. However, standard models like the variational autoencoder (VAE) typically have unimodal Gaussian decoders, which cannot effectively represent the possible se- mantic variations in the space of images. To address this problem, we present new probabilistic generative model called the Hierarchical Bayes Auloencoder (HBAE). The HBAE contains a multimodal decoder in the form of an energy- based model (EBM), instead of the commonly adopted unimodal Gaussian distri- bution. The HBAE can be trained using variational inference, similar to a VAE, to recover latent codes conditioned on inputs. For the decoder, we use an adversarial approximation where conditional generator is trained to match the EBM distri- bution. During inference time, the HBAE consists of two sampling steps: first a latent code for the input is sampled, and then this code is passed to the condi- tional generator to output a stochastic reconstruction. The HBAE is also capable of modeling sets, by inferring a latent code for set of examples, and sampling set members through the multimodal decoder. In both single image and set cases, the decoder generates plausible variations consistent with the input data, and gen- erates realistic unconditional samples. To the best our knowledge, Set-HBAE is the first model that is able generate complex image sets.
INTRODUCTION
Autoencoders are a crucial generative modeling architecture for many applications involving com- plex data, including image (e.g.. Razavi et al. 2019) and scene synthesis (Eslami et al., 2018). image compression (Minnen al. 2018), video prediction (Lee et al., 2018), and model-based reinforcement learning (Ha & Schmidhuber 2018). The variational approach exemplified by the VAE (Kingma Welling 2013) has the benefit of tractable inference and uncertainty estimation while also allowing reasonable control over bottleneck representation and reconstruction quality. While a number of variants have been proposed to incorporate expressive priors for the VAE latent distribution, such mixtures of Gaussians (Dilokthanakul al. (2166) and discrete formulations (Rolfe 2016),, the VAE decoder is very limited given that data dimensions are decoded as condition- ally independent Gaussian distributions. VAE samples therefore tend to be blurry and exhibit little meaningful variation around the predicted mean.
On the other hand, complex multimodal decoder distributions can be learned by alternative genera- tive models, including GANs (Goodfellow et al. (014) and autoregressive models (e.g., PiXelCNN)
van den Oord al. 2016). Conditional formulations of these models (Mirza & Osindero 2014
Van den Oord et al. (2166) allow controlling the sampling behavior to be class or embedding specific. Typically a known class label or metric embedding given when training these conditional models. Conditioning on a class label or embedding allows, for instance in the case of face images, to gener- ate a range of different images of a given person identity varying in pose, lighting, expression, etc., by conditioning on the latent embedding learned by face recognition network (c.g.. (c.g...Van den Oord ct al. (2016). These models can be difficult to train to ensure that the decoder pays attention to the conditioning variable.
Encoder-decoder architectures such as VAE-GAN (aarsen et al. 2015) and BiGAN (Donahue et al.) 2016) improve sample and reconstruction sharpness through their use of adversarial losses. How- ever, none of these models incorporate multimodal decoders into their formulations, and therefore do not lead to diverse sampling behavior.

---

Under review as a conference paper at ICLR 2020
In this work we introduce the Hierarchical Bayes Autoencoder (HBAE). The model incorporates multimodal decoder, in the form of an EBM, which we show under mild assumptions can be trained similarly to a conditional GAN, where the conditioning is the latent code of VAE. We show that the model capable of learning to generate crisp and varied samples. Empirically, the stochastic reconstructions produced by the decoder retain semantic features of the respective input examples as verified by distance in Inception feature space, while unconditionally generated samples are diverse and cover the data manifold, as verified with FID scores (Heusel al. 2017).
We also introduce a simple extension called the Set-HBAE where each data point is collection of examples (e.g,, of images). In this scenario, the traditional VAE no longer applicable, because of the inherent one to many mapping between the latent code and the set elements. We show that Set-HBAE is able be successfully applied to such a setting, and learns to reconstruct input sets in semantically meaningful way, well generate new sets unconditionally by sampling from the latent code prior. Furthermore, we show that the Set-HBAE works well as a method for few-shot classification on ShapeNet (Chang et al. 2015) relative to supervised baseline, especially in the small data regime.
To summarize, the contributions of this work are:
A novel probabilistic generative model trained using variational inference, with a flexible learned multimodal decoder in the form of an EBM.
Empirical results showing that the multimodal decoder is able to produce stochastic reconstruc- tions varying around the local data manifold of examples, and diverse unconditional samples.
An extension of the HBAE formulation model sets of inputs, allowing finer degree of semantic control over sampling behavior, and new method for few-shot classification of sets.
VAE
HBAE (Ours)
Sel-HBAE (Ours)
N
Figure 1: Left: Graphical model for VAE for sampling N data points, where each sample x is con- ditioned on a latent code sampled from a prior, and data dimensions X1 X2 X3 are conditionally independent given z. Middle: In the HBAE, the decoder distribution p=(x(z) is replaced by deep EBM, where data dimensions can interact via undirected connections. Right: In the Set-HBAE, we first sample N set codes z, and then sample M set elements by repeatedly generating from the EBM conditioned on the set latent code.
cGAN
BIGAN
VAE-GAN
HBAE (Durs)
z'
Z
X
Z,X
D
D
Z,X
Z2
X
E
X
Z
x
Figure 2: Architectures for different models. For each architecture, we use same colors to group together modules that contribute cooperating terms to the adversarial loss formulation. The HBAE can be interpreted as cGAN with VAE encoder where all modules are learned jointly.
2

---

Under review as a conference paper at ICLR 2020
2
METHODOLOGY
2.1 VAES
represent family of deep directed graphical models. Given an input x E Ex a VAE formulates a joint distribution of x and a latent variable z E R' as
VAEs (Kingma & Welling 2013)
px z) po (x(2)po(x(z),
(1)
where I denotes the set of trainable parameters; [z( and pe (x(z) denote the prior and decoder distribution, respectively. As common design choice, pa (z) is usually set simple low dimen- sional distribution (e.g., g., an Isotropic Gaussian) which can be easily sampled from; pe (x|z), referred to the decoder, is parameterized identity covariance matrix. This popular instantiation of VAEs offers simple objective function regularized autoencoder where reconstructions are penalized with & loss.
Gaussian distribution with learned mean μlzz; 8) and an
which corresponds
We argue that such a unimodal decoder is not optimal for at least two scenarios. Firstly, in the case where the dimensionality of is much lower than the inherent dimensionality of the input x, a unimodal decoder becomes the bottleneck of learning. In an extreme case, if the latent space is limited to contain only &: bits, e.g., categorical VAE as in Jang al. (2016) with 2 latent units, with & being a small quantity, the model is forced to assign multiple inputs to any given latent code z. This makes having multimodal decoder essential to effectively fit the data distribution. Secondly, for settings where the input does not have vector representation, such as set inputs (Zaheer et al., 2017), not clear how the unimodal decoder can be applied.
2.2 HIERARCHICAL BAYES AUTOENCODER
We now present the Hierarchical Bayes Autoencoder (HBAE), which addresses the drawbacks the VAE as discussed previously. The key insight is design multimodal decoder that flexibily model the implicit (and explicit in the set case) one-to-many mapping from a latent code to target samples. To do so, we replace the decoder distribution of a VAE with EBM-based decoder et al. (2066).
Formally, we start by using the joint distribution in Equation We then define ps(x|z)
[_-"<cxxx)))d> with E(x|z; 0) being the energy function. From the graphical model perspective, this formulation removes the conditional independence assumption between the dimensions of given and allows for modeling rich interactions between them instead. This enables the decoder to assign high probability to multiple possible outputs, if needed (refer to Figure 2 for an illustra- tion). Note that the EBM formulation of the decoder is strict generalization of Gaussian and other exponential family energy-based based models; see LeCun et al. (2006) for more explanations.
During training, similarly apply variational distribution (z(x)) as in VAE, which leads to the following variational objective:
(2/x) log2a(x(2) + min minn Exxxxexaaaaaaa (x) (2)x))
[po[))]
0.0
-Ex-paata(0)) (x)] (z) flog e E(x|2;0)
- KT(gg(Z)x))lpoo())
(2)
-Ex (2vgg (E(|z;; log / -(x(x)0))) dx} + K[[(o(z|x))lnaa(]]]]
where the first row is the standard variational lower bound of a VAE, and the following two rows are the result of plugging in the EBM defination of the decoder.
Directly minimizing Equation 2 as an objective function is difficult as it involves the logarithm of turn to recent connections between deep EBMs and GANs
partition function. As a solution,
(Zhao et al., 2016 LeCun et al. 2006; Wang Liu 2016 Zhai et al. 2016). As brief recap. fol- write its negative log likelihood
lowing Zhai et al. (206)), for an EBM with energy F
3

---

Under review as a conference paper at ICLR 2020
(NLL) as: Ex-ppsnnxx)) [E(x; 0)] logisx e- =F(x()) dx), which can be further developed as:
Exo-penaa (x) [F(x; log { (x())) e F((x:)
F((x0))
dx Exxpaare (x) (F((x; + log (x) gw(x)
(x)
2 > EExxpanna (x) [[ 0)] Ex (x) log e E(x:0)
Exmpeana (x))
(xxx [E(x; Ex (x) (B(;; + H H(g))
qo (x)
(3)
where qe (x) is an auxiliary distribution with parameters v. with ((g)) denoting its entropy. We then let (xx; 6) - Ex; 6) and qu (x)x fxx Ps (z'))))((z) 16) 1)) (i.e., an implicit distribution defined by generator G and known prior pa (z')), which leads to an optimization procedure as follows:
min max Exc-pa... 8)] + Ex~ne (D(( e) 0)] H
(4)
In practice, we rely on implicit entropy regularizers such as batch normalization (loffe & Szegedy 2015 in place of the entropy term H (gv). which gives us a learning objective:
min max Fxx-pasna (x) [xx: 0)]
E (e) 0)].
(5)
which be essentially implemented as GAN, in particular WGAN (Arjovsky et al. 2017 with D being the discriminator and G being the generator.
We then accordingly apply the same derivation and change of variables Equation 2 and obtain practical objective for HBAE:
min max Pxxopaata (x)
D(G PiGl (4)
8,0
(6)
Here both the discriminator D and generator G' are conditional z, which is sample from the encoder 4s- The additional noise distribution Pv. (z'(zz) can also be conditional, but we set to a parameter free Gaussian distribution in practice. Furthermore, we can also extend the objective above in a similar way to other VAE variants such as Higgins et al. (2017); ; Zhao et al. (2017); weighted distance measure
Tolstikhin al. (2017) where the KL divergence term is generalized between the aggregated posterior distribution (z( = Ex-peste (x) 9x) z/xx and the prior pe (z)z, which gives rise to our final objective:
min max Ex Ex-pssta (x) [Ex=qs(x)x)) (xx -) (x D(G(z D(GGE +6
Ex
where D denotes distance between distributions, which we choose the MMD distance as in Tol- stikhin et al. (2017). is a regularization factor.
The training procedure of HBAE is similar to that of a conditional GAN, except that the conditioning variable learned jointly. To be concrete, given a fixed encoder Ja and discriminator D, the condi- tional generator G is updated in the inner loop to increase the conditional discriminator's 's score on G's output. the outer loop, the encoder 9a and discriminator are jointly updated adversarially against G. During inference, we can sample from the generator as with other generative models. However, in a HBAF, sampling follows two step hierarchy. One first sample a latent variable either from the encoder distribution qa (z(x), from the prior pz(z). One then samples from secondary ~ Py (z' to generate an example. This sampling procedure resembles the inference process of a Hierarchical Bayes model, and hence name it Hierarchical Bayes Autoencoder.
2.3 SET-HBAE
Given the HBAE formulation, it is also casy to extend the same generative modeling setting to cases where inputs are sets of images (e.g.. different views of the same object). We refer to this variant as the Set-HBAE. In order to do so, we explicitly express the input in the set format x = (xl x2 ..... x*.... where & denotes the size of the set. By construction, let the elements within set x to be conditionally independent given the set latent code z (see Figure []. We then accordingly modify the HBAE objective Equation aa:
min maxE maxEx-pesc
Ex-ppara (x) [Ezag(tix))
(xx* (2:6) + Ex D(G(z' (z; 4) (2);;
6,0 +
X
(8)
x'Ex
+ (zD(ga(z)(pooz)))
4

---

Under review as a conference paper at ICLR 2020
where xx] denotes the size of the set. The encoder qe in this case takes form of set encoder. Here adopt the DeepSet (Zaheer et al. 2017) mechanism to average pool the codes of elements within the set; the form of the generator remains unchanged. During inference time, the hierarchical sampling procedure naturally corresponds to sampling a set code z first, followed by sampling an element code z'. In this way, the Set-HBAE is able to perform both stochastic reconstruction, and well as unconditional set generation.
3 RELATED WORK
Autoencoding generative models. The HBAE is similar to a handful of existing works which in- volve an encoder, decoder and a discriminator. Notably, BiGAN (Donahue et al. 2016) trains a two channel discriminator that tries to separate the two joint distributions p(z)p(x(z)) 2)) and p(x)p(z|xx) which involves encoder and decoder, respectively. BiGAN differs from the HBAE in that the mapping between z and x is unimodal. Also, in a BiGAN, the encoder and decoder work jointly against the discriminator, whereas in the HBAE, the encoder and the discriminator work jointly against the generator. The VAE-GAN (Larsen et al.] 2015) is a VAE/GAN hybrid model that con- sists of the three modules. In addition the deocder being unimodal, a VAE-GAN deploys an unconditional discriminator, which contrasts with the conditional discriminator used in HBAE
Conditional GANs. A conditional GAN (cGAN) (Mirza & Osindero, 2014 Miyato Koyama 2018) modifies the generator and discriminator of a GAN to be conditional on class labels. The HBAE can be considered generalization of cGANs, jointly with the generator and discriminator. To this end, it is possible to come up with semi- supervised learning version of HBAE where a small portion of input data have observed labels, which are jointly trained with unlabelled data. Refer to Figure 2 for a comparison of the HBAE with the cGAN, BiGAN and VAE-GAN architectures.
it learns the conditioning representation
PixelCNN-based multimodal decoders. Another line of work that employs a multimodal encoder represented by PixeIVAE (Gulrajani et al. Oord et al.) 2016). While similar in spirit to work, PixeICNN suffers from capturing long term dependency of images, and is also expensive during inference time. We see HBAE as providing an alternative approach to this line of work, which leverages the power of state of the art GANs for decoding.
| 20T6). which uses a PixeICNN-basee decoder (Van den
VAEs with large latent codes. Another design philosophy of VAEs is represented by VQ-VAE2 (Razavi et al., 2019), where large latent code is used for each example, with preserved spatial dimensions. VQ-VAF2 then learns powerful prior in the form of a PixelCNN, which is able to generate good quality samples at inference time. We note that in this scenario, the use of a unimodal decoder is well justified where the latent space is large enough to-one mapping between latent code and an input. The HBAE on the other hand, works best in the setup where a compact latent code is used, or in applications where explicit multimodality is desired, such as with sets as inputs.
allow learning a one-
Generative Modeling of sets. Generative modeling of sets is a challenging task. Hierachical Bayes models have previously been widely applied to document modeling. which can be considered as set of words, represented by the LDA model (Blei et al. 2003). However, due to the discrete nature of words, LDA only needs to rely
hierarchy of simple distributions such as the Dirichlet and multinomial distributions. Recently, there has been a a body of work dedicated to point cloud gener- ative modeling, where sets are composed of 3D points in Euclidean space (Li et al. 2018 Achlioptas eet al. 2017). However, sets considered in our work contain much richer structure than point clouds, which contain very little structure per element. It is thus unclear how methods developed from the point cloud or document modeling lines of work generalize to more complex data.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 ARCHITECTURE
The HBAE defined in Equation 7 has three trainable modules: an encoder which corresponds to [z(2]x] [ conditional discriminator (xx[2;0) 0), and conditional generator G(z' z; v)). Our archi- tecture design largely mimics that of a conditional GAN, particular the projection-bad saasedd discrim- inator proposed in Miyato & Koyama (0118). Here the latent code, takes place of the label, y, in a conditional GAN, and is jointly learned with other modules instead of being given. We follow simple design choice for the architectures, where both the encoder and discriminator consist of stacks residual blocks with Spectral Normalization MMiyato et al.] 2018). and the generator con- sists of stacks of residual blocks with Batch Normalization (loffe & Szegedy 2015). Refer to Figure 3 for more details.
Encoder
Generator
Discriminator
512
3a84x84
X
Soore
1
1
Linear
3a84084
Conv
DatProd
1
f hpg Z
256
GlAvgPool
PM
ResBlock 12
BxBx256
ResBlock
250622332
ResBlock 12
512
Linear
t
8x8o258
ResBlock
258x18x18
ResBlock
259
GlAugPool
BeBicas
ResBlock 12
250atke
ResBlock 12
Bxax256
ResBlock
16x10u256
ResBlock 42
200wbxt
Linear
Beta25
ResBlock
1
32X50
ReeBlock 42
644
Concat
Beb2260
ResBlock 12
512+32
0x256
ResBlock
ResBlock
M488433
X X
Figure 3: HBAE implementation. Left: Encoder. Middle: Conditional generator. Right: Conditional discriminator, which takes as input ground truth x or sample 8. The discriminator score is dot product of the intermediate representation and the latent code D(xlz) zx))*z.
4.2 TRAINING PROTOCOL
In all the experiments, follow the training protocols of standard GANs in Donahue et al. (016)), where the encoder and discriminator are trained jointly against the generator, as shown in Equation [7] We use Adam with a learning rate 2e-4 for the generator and learning rate 4e- 4 for the encoder and discriminator. The moment terms are set as O for both optimizers. In practice, one failure mode we have found by directly following Equation 7 is that the encoder can collapse to a constant mapping which makes HBVE regress to an unconditional GAN, As remedy, we put an additional reconstruction regularization term on the encoder's parameters R(Ø) AEx (x)x) ||x G(()2)]l (for the Set- HBAE, this corresponds to mapping to all elements within the set: XEx (x)) (x)x)(( 1
[x] Ex'xx ||x' G(O)z)]l1). Here À is the regularization strength which we set as We note that this regularization term not essential for HBAE to train, but in most cases helps with the stability.
4.3 SINGLE IMAGE INPUTS
In the first set of experiments, we test the HBAE in the standard setting where inputs are single images. We the CelebA Dataset (Liu al. 2015) with resolution 64 64. In Figure we show some of the qualitative results. We see that the HBAE is able to reconstruct input examples in
In our implementations, we adopt deterministic encoder which corresponds to setting do a 8 distribu- tion, which gives the most stable results.

---

Under review as a conference paper at ICLR 2020
Figure 4: Reconstructions and samples from HBAE. Left: input images from the test set (first row) and four stochastic reconstructions with different (second through last rows). Right: unconditional samples (first row) and four stochastic samples with shared and different z'.
Figure 5: Markov chains starting from real examples from the test set (top row) and 6 recursive consecutive stochastic samples (second through last rows). We two consecutive Markov chains staring from the same examples, corresponding to columns the left and right plots.
semantically meaningful way by preserving certain aspects of the input face, while outputting varia- tions that exhibit rich and changing details, such as adding or removing glasses, varying expressions subtly, or changing hairstyles. When sampled unconditionally. the HBAE is also able to produce sharp examples. As control experiments, we have accordingly trained a GAN and VAE using the same architectures (encoder and generator), and reported the FID score of the unconditional gener- ations, and reconstructions (when applicable), in Table We have also included the reconstruction MSE of the VAE and HBAE evaluated in the pixel space and Inception V3's feature space. Refer to the caption for detailed discussions.
Additionally, we show that the HBAE can be used to sample images in a Markov chain, exploring the data manifold around an example as a random walk. The Markov chain starts from real example, and recursively passes it through the encoder and the generator. We show two such chains in Figure 5) starting from the same test examples, both of which exhibit smooth transitions from one sample the next, without burning into single mode. Interestingly, the two chains also lead to different sampling paths, due to the multimodality of the generator. This behavior is not possible with regular VAE or GAN formulations.
4.4 SET OF IMAGES AS INPUT
In the second set of experiments, we change the inputs to be set of images. The first dataset we use is the ShapeNet dataset from Chang et al. (2015), which consists of 3d objects with projected 2d views. We use a subset which contains 13 popular categories. namely airplane, bench, cabinet, car, chair, monitor, lamp, speaker, gun, couch, table, phone, ship. The training set consists of 32,837

---

Under review as a conference paper at ICLR 2020
Table 1: FID scores together with reconstruction MSE evaluated in pixel space and Inception's is fea- ture space. In a control experiment setting, HBAE achieves significantly better FID scores than the VAE counterpart, while being more comparable to unconditionally trained GANs. As a reference, the FID score for ground truth test data is 4.08. Next, reconstruction MSE for the HBAE is signifi- cantly larger than the VAE counterpart evaluated in pixel space, as expected given that the decoder is multimodal and stochastic. However, when evaluated in Inception's s feature space, HBAE and VAE metries are comparable. As reference, the Inception MSE for random pair of test images 0. 12..
Model
FID recon
FID gen
MSE Inception
MSE pixel
HBAE (ours)
12.08
16.72
0.076
0.105
VAE
44.57
65.97
0.071
0.019
GAN
9.13
Table Few shot classification results on ShapeNet (Chang et al. 2015). We show the test accuracy of each features while varying the number of training examples. Here random denotes uninitialized features with the same architecture, and supervised denotes the supervised training setting with the same encoder architecture.
# Training example
64
128
256
512
1024
2048
Set-HBAE (ours)
50.5
56.0
65.1
70.8
74.0
76.2
VAE
39.2
49.9
56.3
60.0
60.7
62.9
Random
19.2
19.2
19.2
19.2
19.2
19.2
Supervised
44.7
52.2
60.5
64.4
70.2
76.3
Figure 6: Reconstructions of the ShapeNet (left figure) and VGGFace2 (right figure) daatasets with a Set-HBAE, For each block, the first row corresponds to an input set from the test split, and the second row shows one (stochastic) reconstruction of the set from the Set-HBAE.
8

---

Under review as a conference paper at ICLR 2020
unique objects and 788,088 images in total. The second dataset is the VGGFace2 dataset from Cao et al. (2018), which a face dataset containing 9,131 subjects and 3.31M images.
For both datasets, randomly construct fixed-size sets of elements. For ShapeNet, this is done by randomly selecting an object and of its views; for VGGFace2, we similarly select one identity and 8 of her/his faces. A mini-batch is constructed by selecting N (which we to 32) objects/Subjects, which amounts to N images. Example set inputs together with their stochastic reconstructions are shown in Figure The qualitative results indicate that the samples are diverse, yet consistent with each other and tend to resemble the semantic traits of their respective input sets (i.e., same object in different views for the ShapeNet case and same identity with varying attributes for the VGGFace2 case).
Figure 7: Uncurated unconditional generations from Set-HBAE trained on ShapeNet (top) and VGGFace2 (bottom) with size fixed to 8. Fach column corresponds sampled set. The Set-HBAE generates sets that are coherent, and often with good visual quality.
As further quantitative measure, we also conducted few shot classification experiments on ShapeNet with the learned features. We compare the features learned from our Set-HBAE with control VAE, well as with randomly initialized features, and with supervised training results, all with the same encoder architecture. For all settings, we trained linear classifier with crossentropy loss, without any explicit data augmentation or regularization (except for the supervised learning set- ting where the encoder is jointly trained with the linear classifier). We vary the number of training images from 64 to 2048, and report the results in Table 2 We see that in the few shot classification domain, the learned features derived from the Set-HBAE largely outperform the supervised learning approach as well as the VAE results, all which far exceed random features.
Finally, show the Set-HBAE's ability generate unconditional samples. The generated results are shown in Figure forr both ShapeNet and VGGFace2. Notably, the Set-HBAE able to generate diverse sets, where elements within the generated sets also diverse but consistent in ways that preserves object shape or face attributes. To the best of our knowledge, this is the first generative model that is able generate complex sets with richly varying yet coherent structure.
CONCLUSION
In this paper, we presented the HBAE, novel probabilistic generative model with EBM-based multimodal decoder. In our implementation, we jointly trained the encoder with variational infer- ence and the EBM decoder with powerful conditional GAN. We demonstrated that the decoder capable of generating diverse conditional and unconditional samples. Then we introduced the Set-HBAE,, novel generative model for sets of examples, generalizing the HBAE framework. The Set-HBAE takes sets as input and can sample plausible variations that are consistent with the con- ditioning set, but generalize beyond the specific input instances. In addition, show that the

---

Under review as a conference paper at ICLR 2020
Set-HBAE performs well ShapeNet Chang al. (2015) few-shot classification task, show- ing the advantage of leveraging generative models in the small data regime. The ability to sample plausible variations conditioned on imput example or set of examples is a key building block for enabling predictive models, for which we see future applications in areas such as video (Lee et al.) 2018), 3D scene generation (Eslami et al. 2018), and model-based reinforcement learning (Ha & Schmidhuber 2018).
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. arXiv preprint arXiv:/707.02333 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 17X1v11701.0785. 2017.
Wasserstein gan.
arXiv preprint
David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 39Jan):933-102 2003.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67-74. IEEE, 2018.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012. 2015.
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Gamelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders, 2016.
Jeff Donahue, Philipp Krähenbühh, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv:1605.0990 2016.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frédéric Besse, Fabio Viola, Ari S. Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil C. Rabinowitz, Helen King, Chloe Hillier, Matthew M Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360: 1600:1044 1204-1210,, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. Pixelvae: latent variable model for natural images. arXiv preprint 2016.
David Ha and Jürgen Schmidhuber. World models. CoRR, abs/1803.10012. 2018 URL http:
7/arxiv. wwwpi//..rr/bb///0..03 10122
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems 30, pp. 6626-6637. Curran Associates, Inc., 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. 2017. URL https://cpenreww www.:oottttttte.o http://wwwww.eeeeereeeeeaa--eee.
10

---

Under review as a conference paper at ICLR 2020
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioon with gumbel-softmax, arXiv
preprint arXiv: 140, 2016.
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114, 2013.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther, Autoen- coding beyond pixels using learned similarity metric. arXiv preprint arXiv:1512.099 510009000,0.. 2015.
Yann LeCun, Sumit Chopra, and Raia Hadsell. A tutorial energy- baasyy learning. 2006.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine.
Stochastic adversarial video prediction, 2018
Chun-Liang Li, Manzil Zaheer, Yang Zhang, Baraabas Poczos, and Ruslan Salakhutdinov. Point cloud gan. arXiv preprint arXiv:1810.05795 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
David Minnen, Johannes Ballé, and George D Toderici. Joint autoregressive and hierarchical pri- for learned image compression. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett (eds.), Advances in Newral Information Processing Systems 31, pp. 10771-10780. Curran Associates, Inc., 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
vrXiv:14111 1784, 2014.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint
arXiv: /802.0567 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: D.XXiv/802.059555 2018.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv: 1906.004000 2019.
Jason Tyler Rolfe. Discrete variational autoencoders, 2016.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. arXiv preprint arXiv:1711.0/555 2017.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelcnn decoders. In Advances neural information processing systems, pp. 4790-4798, 2016.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu, Pixel recurrent neural networks. CoRR, abs/1601.06659, 2016. URLhhtt:://axxii wtgg://s///.....000 06759
Dilin Wang and Qiang Liu. Learning to draw samples: With application to amortized mle for generative adversarial learning. arXiv preprint 161.v:16 2016.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. In Advances in neural information processing systems, pp. 3391-3401, 2017.
Shuangfei Zhai, Yu Cheng. Rogerio Feris, and Zhongfei Zhang. Generative adversarial networks as variational training of energy based models. arXiv preprint arXiv:1611.0179) 2016.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126 2016.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02260 2017.

---

Under review as a conference paper at ICLR 2020
TE
TE
Figure 8: VAE results on the Celeba Dataset. Top left: random samples of Celeba image; top right: reconstructions of a VAF; bottom: unconditional samples from a VAE.
A APPENDIX
12

---

Under review as a conference paper at ICLR 2020
Figure 9: Unconditional GAN results on the Celeba Dataset. Top left: random samples of Celeba
image; top right: reconstructions of GAN; bottom: unconditional samples from GAN. The reconstructions are achieved by first training the unconditional GAN then train an additional encoder with the generator as decoder with & loss.
13


Reviews:
[{"review_id": "rJlN9wthFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work a novel probabilistic generative model is introduced mixing several existing frameworks: The Hierarchical Bayes Autoencoder (HBAE) can be interpreted as a cGAN with a VAE encoder. They also incorporate a flexible learned multimodal decoder in the form of an EBM. The authors claim to produce stochastic reconstructions varying around the local data manifold of examples, and diverse unconditional samples. In addition, they present an extension of their HBAE formulation in order to model sets of inputs. This is one of the main contributions of this work since generative modeling of sets is a challenging and unsolved task. \n\nAlthough there is a clear explanation of the contributions of this paper, the motivation of this study is not precisely described. The introduction provides a good understanding of the topic; however, the authors may wish to provide several examples on the interest of the study. They provide a good description of the existing generative modeling architectures and the applications involved but not mention the importance of their contributions to the real world. \n\nThe proposed formulation seems encouraging thanks to the incorporation of multimodal decoders. The derivation of HBAE is theoretically well justified, as a result, one could replicate or further work in this paper. The authors explain in detail the procedure they took in order to arrive to the final HBAE formulation giving a clear understanding of the topic they present.\n\nBy describing the related works, they give an understandable perspective about the different drawbacks and differences of the existing methods. One could deduce the motivations of this work thanks to this section, however they should have been state clearer at the beginning. \n\nThe authors explain in detail all the different parts of the architecture, presenting precise information about the different layers. At first sight, it seems like one could reproduce the methodology used. It is clearly explained and discussed.\n\nI think the experiments conducted to evaluate this work are not enough. The authors performed several experiments in order to prove the efficiency of their methodology by showing the generated images and investigate the qualitative results. If we look at the results in figure 4, we can observe some artifacts and blur regions in the image. Moreover, the variations on the faces sometimes seem more deformations than a different human feature. It would have been interesting to evaluate the identity preservation to further investigate the quality of the images. Although the methodology, conceptually, suggested an interesting approach, the results are not so encouraging given the quality of the images achieved by other methodologies.\n\nApart from comparing quantitatively the capacity of the generated images with other related works, it would have been interesting to show that qualitatively by showing images from the mention methods. In general, there is some information missing in order to replicate the results and more details would have been appreciated. \n\nRegarding the experiments about the set of images as input, the quality of the images is similar to the simple HBAE. Since this is a more challenging approach and one could not compare with so many existing methodologies, it seems like they are going to the right direction to achieve the desired images. However, looking at the exposed results, the quality of the images is not realistic and has a lot of artifacts. Some other experiments could have been performed in order to show more interesting results.\n\nQuestions:\n\n-\tCould you better explain the meaning of figure 1? I think it was a good idea to exemplify the differences but is not clear enough.\n-\tDid you perform any face alignment when dealing with the different faces databases?\n-\tDid you think about the possibility of giving a concrete condition for varying the image?\n-\tWhich could be some concrete applications for your work?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H8Xg7Ttb9aS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I don't think this paper should be accepted.  In my opinion, the mix of EBM and VAE is not really compelling;  and it is not clear at all to me that one gets much from the \"V\" in this setting.  Furthermore,  the experimental results are not great either qualitatively (by the standards of generative-models-of-images in 2019) or quantitatively (even by the standards of GAN papers).  Finally, the author's claims about sets seems tacked on, and unrelated to the rest of the paper.  Modern neural networks (attention/transformers/graph-nn, etc...) handle sets naturally, and could be used with any other conditional generative model.  To my eye, the results in e.g. figure 6 do not really seem like the model is matching the set, and the authors make no attempt to formalize or quantify how well their model generates things \"in the set\".  \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygOn4QjuH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article proposes to replace the conditional distribution p(x|z) in an VAE by a general graphical model parameterized by a deep neural network. Using a series of approximations it arrives at a triplet of models, where the graphical model is split into a model representing the likelihood and a model that generates samples from the distribution. the third model is the usual autoencoder. The core idea is that the latent space of the VAE serves as bottleneck so that the graphical model develops a multi-modal distribution. At the same time, the use of a neural network allows for a much better modeling than using the typical local Gaussian distribution as is used in most VAE-approaches. Experimental results show that reconstructions and markov-chains starting from reconstructions produce decent results. An extension to sets is proposed as well as a task showing performance for semi-supervised learning.\n\n-------------------------------------------\nThe text above serves mainly as a  summary of what the paper sets out to do. However, we have to check whether the paper is actually achieving this goal. In my view, it does not, and it suffers from severe problems on the theoretical side. Furthermore, important evaluations that are standard within the VAE framework are missing: samples from the true underlying distribution. In the remainder of this review, I will try to substantiate the claims regarding theory which lead me to an overall decision to reject the article in its current form.\n\nLet us start from a birds-eye perspective: the model consists of three parts:\nthe Encoder E, which i will refer to as q(z|x)\nthe Energy-function, also called E(x|z) and with -E referred to as D later on.\na generator function G(z,z') that is supposed to sample from the distribution p(x|z) = 1/Z(z) exp(-E(x|z)) using entropy generated by z'~N(0,I). I will refer to this distribution as r(x|z)\n\nThis split is smart: using a general graphical model to model p(x|z) makes it very hard to sample from the distribution and it is difficult to estimate the normalization constant Z. The paper refers to Zhai et.al (2016) to find eq(3). I really like (3) because the inequality is tight when G is producing samples from p(x|z). To be more exact, we have that the estimator for the normalization constant is log(Z) - KL(r(x|z) || p(x|z) ). This reveals the first issue: the generator must have an r(x|z) > 0 for all x except a countable infinite subset. This is difficult to achieve with dimensionality of z and z' smaller than dimensionality of x and G being deterministic. Therefore, KL(r(x|z) || p(x|z) ) is already infinite. This is a technicality that i will ignore, since we can always add a minimal amount of noise on the output of G, even though this paper does not.\n\nThe real problem starts with eq(4). Since r(x|z) is implicit, there is no way to compute its entropy, which is then just discarded. The paper argues that this is no problem, because they intend to somehow bound the entropy. We will look at this point, later. \n\nThe issue with bounding is that for this to be meaningful, we also need a bound on the entropy of p(x|z), because otherwise there is no way for r(x|z) to achieve a small KL-divergence. If we now look at (4), we can compute the optimal r without bounded entropy. The optimal r is the distribution that by (5) maximizes -E(x|z) for a given z. this result is not a multi-modal distribution but a delta-peak on the optimum and certainly not p(x|z). so the bound is super important and technically we need a bound on log|det(d/dz' G(z,z'))|. \n\nThis is difficult from a theoretical view-point as well as from a modeling view-point because we know that we have to have nearly discontinuous transformations to create sharp edges. Further, to transform z'~N(0,I), into a multi-modal distribution, there must be a function g(z')->w so that w is multi-modal. This is difficult to achieve with bounded determinant.\n\nLooking at the experiments, let's see whether the determinant is actually bounded. The paper claims that bounding can be achieved using batch-normalization. First this requires a reference on why this should be the case and show that this holds as implemented for the resnet block and second, this requires an estimate for how large the entropy can still be. if the entropy can be very large, we can not claim to train a VAE, because the error going from (3) to (4) is large. Secondly, we have to ensure that both E and G have bounded entropy, because otherwise KL(r(x|z) || p(x|z) ) will be large and there is no way to that the approximation in (3) is good, at which point E can not be claimed as doing maximum-likelihood on the data.\n\nAs shown in Figure 3, the Generator has a Conv-Block and the initial layer is a linear layer. Similarly the discriminator (or E) has a linear output layer. Therefore, the entropy is not bounded as the paper does not claim to have any entropy normalization here. The paper does not say what the activation functions are, but if the final layer of E is linear or Relu, we can add yet another problem to the list (next to the unbounded Jacobian): the VAE framework stops making sense. If we look at eq (6) or (7), the final divergence term (multiplied by beta) hinges on the fact that it provides a suitable error contribution. However, if the final layer of D is linear, we can just scale that layer arbitrarily large until the error contribution of the divergence vanishes. \n\nLastly, the Energy-function does not even necessarily model a probability function. Remember, we need that Z(z)=\\int exp(-E(x|z)) dx <\\infty. However, the definition by Figure (3) is E(x|z)=-h(x)^T(z) and the output layer of h is linear. so we can find weights such that h(x)=c \\forall x. In which case \nZ(z)=\\int exp(-c^Tz) dx = exp(-c^Tz) \\int 1 dx \nwhich is only bounded if x is bounded, which I can not see since the generator is not bounded as well. This is only one of many counter-examples.\n\n\nA few other small things:\n- For z from a normal distribution the bottleneck discussion in 2.1 last paragraph does not apply, because for a dataset with N samples it is always possible to partition the real-space of Z into N partitions with probability 1/N each. This even works with 1-dimensional normal distributions, independently of the dimensionality of x. So a multi-modal distribution for p(x|z) is really not necessary if encoder and decoder are strong enough to memorize the dataset.\n- below (5) it is claimed that this is similar to a WGAN, but WGAN have a difficult constraint to be 1-Lipschitz to make it work (which proves to be extremely difficult in practice as it translates to a similar inequality constraint on the jacobian)\n- In Fig(5) it is unclear to me how a markov-chain is defined using the model. I assume this is done by taking samples from z', generating a new x and encoding this to obtain a new z. If z' models different features from z (assuming z' is multi-modal) we would assume that the z would remain the same in each point, so no exploration of p(z) takes place. However, we can already see in the lat rows that the pictures look more strange, so some mixing is taking place but it is not clear how that relates to p(z). Therefore i would really like to see images generated by sampling z~N(0,I) and z'~N(0,I) to see whether the generated images look reasonable. \n- Moreover, to show that the Generator actually produces images that are likely under p(x|z), we would also need samples generated from the true underlying distribution p(x,z), e.g. using hamilton monte-carlo.\n- Finally, i have to notice that the model, while called hierarchival VAE, does not provide a way to compute z' from the input. is that than really an autoencoder?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a probabilistic generative model which mixes a variational autoencoder (VAE) with an energy based model (EBM). As mentioned by all reviewers (i) the motivation of the model is not well justified (ii) experimental results are not convincing enough. In addition (iii) handling sets is not specific to the proposed approach, and thus claims regarding sets should be revised.

Decision: Reject