Localized Generations with Deep Neural Networks for Multi-Scale Structured Datasets
Authors: Yoshihiro Nagano
Shiro Takagi
Yuki Yoshida
Masato Okada
Venue:  (2020)
Keywords: Variational autoencoder
Local learning
Model-agnostic meta-learning
Disentangled representation

Abstract: Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.

Under review as a conference paper at ICLR 2020
LOCALIZED GENERATIONS USING DEEP NEURAL NETWORKS FOR DATASETS WITH STRUCTURAL SIM- ILARITY
Anonymous authors Paper under double-blind review
ABSTRACT
Extracting the hidden structure of the external environment is an essential com- ponent of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the struc- tural relationship between the data points depending on location observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighbor- hood, respectively. However, this approach is only limited to use with simple model, since the complex model, including deep neural networks, requires a mas- sive amount of data and extended training time. In this study, overcome this trade-off based on the insight that the real-world dataset often shares some struc- tural similarity between each neighhorhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder have the dif- ferent model parameters for each local subset and train these local parameters by the gradient- based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.
INTRODUCTION
Extracting the hidden structure of the external environment is essential for achieving intelligent agents and modeling human learning (Kemp & Tenenbaum 2008 Lake et al.) [015] Higgins et al.) 2017 Achille al., (2018: Saxe al. (201)). Human beings and/or animals can effectively learn internal representations from few amounts of experiences. Various methods of nonlinear feature extraction (Maaten & Hinton| 2008 McInnes et al., 2018) are recently proposed to model the com- plex environment. In addition, thanks to the developments of deep generative models (Kingma & Welling 2013; Rezende et al. 2014 Goodfellow et al.) 2014 Rezende Mohamed] 2015). we can now handle the high-dimensional dataset many individual problems.
Although recent studies succeeded modeling the dataset for the specific problems, there are still challenging properties in real-world. The datasets that we are interested in often characterized by the locality: the change in the structural relationship between the data points depending on lo- cation in observation space. For instance, sequence of experiences gradually changes according to multiple aspects, including time, space, and modality; we need to identify each individual during the development of their faces consistently. Besides, the human-min made objects often have multiple color options for the same shape or size. Many studies have incorporated locality for dimensionality reduction and representational learning (Kambhatla & Leen 1997] Roweis & Saul 2000, Tenen- baum et al. 2200). For example, combining the local learning approach with classical unsupervised learning algorithms such as PCA significantly improves their model capacity. These studies aim to find mappings between the data and the coordinate space under the assumption that the data space
composed of multiple oow-dimensional subspaces (Brand) 2003 Vincent & Bengio 2003). We linear projection for embedding models
refer this approach to local learning. They usually and a l distance in the input space for neighborhood construction. In general, we can arbitrarily choose the distance for the neighbor graph, and it affects the quality of the embeddings.

---

Under review as a conference paper at ICLR 2020
(a) Dataset with local structure
(b) Local VAE
Local
Meta parameter
Sample
Local ombedding
Local embedding
Local embedding 3
Structural similarity
X
X
Figure Schematic diagrams of localized generations.
Incorporating the local learning approaches into the training of the deep generative models will give model that has both the capacity for tigh-dimensional inputs and flexibility for locally changing environments at the same time. However, the integration of these two paradigms is not trivial. The conventional local learning approaches train the different embedding model for each neighborhood from scratch, nevertheless the deep neural networks generally require a large amount of data and take long training time (LeCun al. 2015). In other words, local learning approaches learn internal representations for each neighborhood by using only relatively simple models, whereas deep generative models learn one complex representation as whole with deep neural networks.
To overcome this trade-off, the structural similarity between each neighborhood is the key. In the case of the human face, although each face varies greatly depending on age, gender, and etc., there are common facial expressions (Ekman & Keltner 1997). It is reasonable to expect that each local subspace of the dataset shares some structure since most dataset tends to governed by the consis- tent rules of the physical world (Achille et al. (2018). 8). Under the assumptions about the locality and structural similarity, a dataset has two scales structures: the local structure inside some neighbor- hood and the global relationship between each neighborhood. Figure la visualizes these two scales inside the dataset.
When dataset has structural similarity, meta-learning is effective approach. Meta-learning an algorithm to learn the rules for each task quickly for a dataset consisting multiple tasks (Schmid- huber 1987 Bengio et al., 1992. Andrychowicz al. 2016 Ravi & Larochelle, 2017: Finn al. 2017]. In this study, by considering each local neighborhood as task for meta- leatnneg. we extract the transferable knowledge between each local structure. We propose to train the meta embedding model, which parameters capture the common local structure and quickly adapt to each subspace by utilizing the structural similarity.
We generalize the typical deep generative model called the Variational Autoencoder (VAE) (Kingma & Welling 2013 Rezende al. 2014) be applicable to the dataset with local structure. We extend the graphical model of the VAE to have different model parameters for each local subset of the dataset (Figure 1b) while keeping to avoid a large amount of computation by using the gradient- based meta-learning (Finn et al. 2017; Grant et al., 2018). By treating the neighborhood of each data point as task to adopt the meta-learning, we make our proposed Local VAE possible to learn similar structures between neighbors quickly. We evaluate the performance of our proposed model with the 3D Shapes Dataset (Burgess & Kim! 2018) and the concatenated dataset of the Cars3D (Reed et al., (2014) and SmalINORB (LeCun et al.) 2004). The numerical experiments shows that the locality enables the model to achieve the disentangled representation for each subspace without any label information.
BACKGROUND
2.1 VARIATIONAL AUTOENCODER
First, we introduce the VAE Kingma Welling 2013; Rezende et al., 2014), which is one of the deep generative models that have been studied extensively in recent years. The objective function
2

---

Under review as a conference paper at ICLR 2020
of the VAE is defined the variational lower bound of the log-like rigllikeihhood (referred as the evidence lower bound, ELBO) for the dataset. Given dataset D = {ax()) ) /== each (+);
ELBO is defined as follows for
log (p(x)(()))))))))))))()) 2 LE (o( log pe (2)
DKL ( po(x)) -C(0, xx)), (1)
where po (xxx(())) is the conditional likelihood referred to as the decoder, and (=(=(x(0)) is the variational posterior distribution referred to the encoder. The choice of the prior Pe is typically the standard normal, and the posterior distribution also variationally approximated by a Gaussian. This parametric formulation of 46 is called the reparameterization trick and enables the evaluation of the gradient of the objective function with respect to the network parameters. Overall, we can train the decoder and the encoder networks by taking the minimum of the negative ELBO using the gradient descent method.
2.2 MODEL-AGNOSTIC META-LEARNING
Then, to incorporate the local learning approach into the VAE, we utilize the Model-Agnostic Meta- Learnnng (MAML) (Finn et al. 2017), which is gradient- based meta- meta-learning algorithm. The goal of MAML to find tusk-independent knowledge from a number of previous related tasks. Once the meta-learne learns the task inse-independent knowledge, it can quickly adapt to a new task using only few data points and training iterations. For connection to the deep generative models, we introduce the setting based on the maximum likelihood estimation described in Grant et al. (2018) instead of the original MAML formulation. In the setting of MAML, each data point is assumed to be sampled (x). The MAML objective function in from the task specifce distribution (f1) ....,.". (ix) ..x,, maximum likelihood setting is
((
Σ
Σ
logo x ((xtm))
V V
1 K
Σ log pl (n))
10)))1))
(2)
N
K
where 8',, is the task-specific parameters after a single batch update by gradient descent from 6. The meta-learner achieve the parameter 9, which quickly adapt new tasks with small amount of data by optimizing [Equation 2 using the gradient method. We note that O can be inter- preted the parameters of the prior distribution for the task-specific parameters 0 By replac- ing the expectation w.r.t. the original posterior distribution by the maximum likelihood estimate ((0)))(070))d) f(O')). ). the abovementioned objective function (Equation 2 recovers.
LOCAL VARIATIONAL AUTOENCODER
In this section, we will present the Local VAE, variant of the VAE suitable for representation learning of a dataset with local structure.
Here, we extend the objective function of the VAE [Equation 1 to have different parameters for each local subset. We consider the variational lower bound the log-likelihood for the dataset D, just with the Vanilla VAE. However, we define the different model parameters 0 ((((()) and (x())) for each neighborhood (*) of the i-th data, respectively. Since these parameters are often high- dimensional and require long time and large amount of data for training, give meta parameters and & as prior distributions these local parameters. The overall model performs the probabilis- tic inference through the conditional distribution from the meta parameters. The variational lower
3

---

Under review as a conference paper at ICLR 2020
bound for the log- leeliiklihood can be calculated as follows:
log (x (x) = log
p(z
((at))) (0)co)000/dddd (0)
(3)
/ q(z (")
(2( ((
p(x N (()to)) (0)
log
dzd0 Niaoo
(4)
((2/=10)
(
g(=|=()) 2 xxx
(+)
logp(w z ( N
DKL ((z (1) ( (
(5)
where p(0 (a) (( 0) and (()) (6) are the conditional distribution of the local parameters. We note that the integral variables of the expectation and the Kullback-Leible divergence in Equation are O N( and ()))
As we mentioned above, the integral variables of Equation 5 include O (((())) and (()())) This means that [qquation needd to take an integral of the model parameters to evaluate the objective function, while the one of Vanilla VAE only requires the Monte Carlo expectation of Such an inte- gral is unreasonable in deep generative models where model parameters are often high-dimensional. To overcome this problem, we replace this integral with the maximum likelihood estimator updated by the one-step gradient method, we described in Section 2.2 Let L(O, Φb; ((() be the negative of the expression obtained by Equation 5 By replacing the integral of & (((()) and (N(())))) with the maximum likelihood estimator 0 NNaato) and (iv(ata)) obtain
C(O, Φ;; x (1)) - - senttyy logp(<>(0) O ( + DKI. [((2/z()) ( po(z)
(0 ( (0))
(6)
We note that the integral variable of Equation 6 is now only z. The maximum likelihood estimator of the local parameters can be obtained by the following update rule:
(zoti) = - V K
Σ
C(O, Φ x).
xEN(x(()
Φ ('((2 =Ø
L(O, Φ;
NEN((
where K is the number of neighborhoods for (i), L(O, di x)) in the above equations is the ELBO Algorithm shows the overall algorithm.
of Vanilla VAE defined in Equation |
From the perspective of the graphical model, our proposed Local VAE algorithm corresponds to the assumption that the dataset approximately lies on multiple subsets and that each subset is generated objective function
from different parameters. Alternatively, from the viewpoint of meta-learning consistent with the case of training VAE by MAMI when task information given as neighbor graph. We can also give the relationship of our model to the conventional local learning approach. Please see Appendix A for the detail.
3.1 NEIGHBORHOOD CONSTRUCTION
As we mentioned above, local learning approaches have to construct a neighbor graph before training the model. The conventional approaches often the k-nearest neighbor graph build the original data space. In general, we can make arbitrarily choice how to construct the neighborhood, and affects the quality of the embeddings. We evaluated two types of neighborhood in the following experiments: synthetic neighborhood by sampling and k-nearest neighborhood on latent space. In the synthetic neighborhood by sampling, we sampled different examples for each x( from the noise distribution assumed the observation process of the data and used these examples as
4

---

Under review as a conference paper at ICLR 2020
Algorithm Optimization of Local VAïs
1: while until converge do for (9) in mini-batch B do
2:
Sample K -poonts from the neighborhood of x0); x(1), ,..,,,,."""" x(() (xx
4:
Evaluate the local objective L(O, o; Φ; for the K-neighborhood w.r.t the meta param-
eters based on Equation 1 Update the local parameters:
5:
(((())) - ee E EeEN( [(,, Φ; 2))
(a)(() Φ V LaeN(a L(O, Φ; æ))
6:
Evaluate the global objective Cy N(a(1)) & (((()))) x0) for i-th data w.r.t. the local parameters based on Equation 6 end for
7:
8:
Update the meta parameters:
0 - 0 1)
Ve B Eier Eiesfy (O (N(())) (1)),
Φ - Φ V e B EieB (g ((n())) (a (((()))) x (+)).
9: end while
the neighborhood of (*), We considered that this method is effective when the data is densely distributed in the observation space and used this method for 3D Shapes Dataset to omitting the time construct the neighbor graph for the large dataset. On the other hand, in the k-nearest neighborhood on latent space, we used the k-nearest neighbor graph builds on the latent space of the VAE. We can expect that we can obtain neighborhood that follows our intuition by using the distance on latent space rather than the input space (Caron et al. 2019). In the experiment on the CarsNORB Dataset, which we will describe later, used Faiss (Johnson al. 2017) for similarity search and continuously updated the latent code for each iteration during the training phase.
4 RELATED WORKS
In this study, we employed the maadient-baseed meta-learning method MAML (Finn et al., (017) and its probabilistic formulation (Grant et al., 2018) to find local parameters from few data points. Recently, several studies (Hsu al., 2019) Metz al. 2019) proposed the integration of unsuper- vised learning and meta-learning from another perspective. Hsu et al. (2019) proposed the algorithm for generating MAML task information by utilizing embedded similarity information created with unsupervised learning. In contrast to this case of using unsupervised learning for meta-learning, used meta-learning to perform unsupervised learning. In addition, [Mtz et al. (2019) proposed a way seek the objective function itself for representation learning with meta-learning.
The local learning approaches, including LLE (Roweis & Saul 2000) and Isomap (Tenenbaum et al,, 2000), are deeply related our work. We discuss the relationship between LLE and our work in Appendix A in detail. Besides, the extension of generative models to make them applicable for structured datasets has recently been extensively studied. The generalization
the latent space of
the VAE to a non-Euclidean space such as a spherical surface (Davidson et al., 2018), hyperbolic
space (Ovinnikov) 2019 Nagano al. 2019 Mathieu et
2019). or discrete space (Jang et al.,
2017 Rolfe 2017) was proposed.
The property of disentanglement has attracted notable attention in structure extraction using VAEs as described above (Higgins et al. 2017 Burgess et al.] 2017]] Kim & Mnih 2018 Chen et al.) 2018 Kumar et al., 2018 Locatello al. 2019). Most the proposed models try to realize dis- entanglement representation by modifying the penalty term of the objective function network architectures. On the other hand, our approach focuses on how to learn parameters suitable for dis- entangled (local) representations that our proposed method at the same time.
utilize both these aforementioned techniques and tional generation is commonly practiced (Kingma et al.) 2014, (Shhn et al. 2013 MMrza & 2014). Our method is similar these approaches in that the density function is conditioned on the
From the viewpoint of generating data by deep generative model with some supervision, condi-
Osindero
5

---

Under review as a conference paper at ICLR 2020
(a) 0 (Vanilla)
(b)
le-3
(c)
Figure 2: Qualitative evaluation of the randomly-selected conditional prior samples.
neighborhood of the specific data point. While conventional conditional generation generates data by only one model parameter with the known class label as additional latent code, our proposed model has different network parameters for each neighborhood. Moreover, our approach more applicable than conventional methods since our approach does not need any class label information.
5 NUMERICAL EVALUATIONS
5.1 3D SHAPES DATASET
Here, numerically evaluate the performance of the Local VAE. We the 3D Shapes Dataset (Burgess & Kim 2018), which has a clear disentangled property. The disentangled property can be interpreted the simplest case of the local structure. The disentangled dataset assumed be able to control by small number of factors. Since these factors alter the observation in data space, and the scale of them different one by one, we can interpret the factors which significantly affect the observation as the global features and other factors as the local features.
We followed all the experimental settings in Locatello al.) (2019), except the batch size and the number of tasks, to eliminate effects outside the proposed method as much as possible. Please see Appendix B for the detail. We qualitatively assess the generated images and quantitatively evaluate the model performance by using the disentanglement metric (DCI scores) proposed by FEstwood & Williams 2018 and the Fréchet Inception Distance (FID) (Heusel et 2017).
Figure 2 shows the conditionally generated images of the trained models. At the inference phase, the model obtains the local parameters O (( by applying one-step gradient descent using the randomly selected training data and generates images from these local parameters. We trained mul- tiple models with different values of a, which is the hyperparameter
Local VAEs. Note that the original objective function of Vanilla VAEs recovers in the case of since the local parameters are strictly consistent with the meta parameters. According to the subjective assessment, the quality of generated images is better when a is large.
There could be a concern that overfitting caused the result above. If the model obtains local param- that perfectly generate only the training sample be referenced, the quality of the generated image will be superficially high. To exclude this possibility, we visualized the reference samples and their corresponding generated images of the model with a in Figure 3) The lefumost column shows the reference training samples used for the conditional localized generations. Each row visu- alizes the generated images conditioned on the reference sample the left column. We randomly picked ten latent codes z ..... zz from the prior distribution, and then used these codes for every conditional generation. In other words, the images shown in the same column share their latent code. According to the figure, the trained model generated clearly different images in the same row conditioned on one training sample. This result strongly suggests that the Local VAE model did not overfit to the specific data. Moreover, the shape, angle, and size of the object were the same, and only the color was different in each column. These results suggest that the model trained by the proposed method segregated color information global features and other information as local

---

Under review as a conference paper at ICLR 2020
Conditional generations
z
ZZ
z00
Reference samples
Figure 3: The reference training samples and their corresponding generated images of the Local VAE with a =1. The leftmost column shows the reference training samples. Each row visualizes the generated images conditioned the reference sample. The images shown in the column share their latent code. The trained model extracted the color information as the global features and other information as the local features.
features, and obtained an internal representation independent of the global features. Note that Local VAEs only use neighborhood relationships and do not use any label information.
Then, we qualitatively evaluated the latent representations of Local VAEs by using the DCI scores (Bastwood & Williams/ 2018). The DCI scores quantify the learned representations based three types of aspects: Disentanglement, Compactness, and Informativeness. All metrics can be computed from the importance of each dimension of the latent space for predicting a factor of variation. DCI scores require the label information of the ground truth. Since the Local VAE clearly extracted the color information as the global feature, we calculated DCI scores for the two types of conditions: the class labels including all six aspects (w/ Color condition), and the class labels excluding color information (w/o Color condition).
Table shows the empirical evaluations of the DCI scores. DCI scores with six labels (w/ Color conditioon) of the Local VAE were slightly better than the one for the Vanilla VAE. The model with small a, which is the value closer to the Vanilla VAE, tended to achieve better scores for all DCI metrics in the Local VAE comparison. This result is attributed to the loss color information from the internal representation the DCI scores the condition without color. All DCI metrics took their maximum value at a = 1. The performance was slightly degraded 1e2. The numerical evaluation suggests that a can control how much of the structure behind the entire dataset is considered global variation and from where it is regarded as a local variation. We also evaluated the performance of the 3- VAE (Higgins et al. 2017) as reference. The 3-VAE modifies the KL term (Equation 1p by multiplying non zero coeffecint 3. Although the 8- B-VAE with non-zero 3 = 8 or 3 16 achieved higher scores than the Local VAE in the condition with color, the Local VAE with a significantly outperformed all the 3-VAE in the condition without color.
increases. On the other hand, the Local VABs significantly improved
lel, and the loss diverged during training
We also evaluated the quality of the generated images with the Fréchet Inception Distance (FID) (Heusel et al. 2017). FID is a metric that evaluates the similarity of quality between real and generated images. We used the 50,000 samples of the ground truth dataset and generated images for FID calculation. According to Table FID tended to low at the large a and took the minimum value 1. This result consistent with the DCI scores of the condition without color. Since the method to calculate neighborhood arbitrary, we also compared the synthetic neighborhood with widely used approach. Please see Appendix for the detail.

---

Under review as a conference paper at ICLR 2020
Table I: Quantitative evaluations of the Local VAE on the 3D Shapes dataset. Highlighted cells indicate the model with the highest performance in the comparison of Local VAEs. Bold numbers indicate absolute best results.
DCI w/ Color
DCI w/o Color
FID
Disent.
Compl.
Inform.
Disent.
Compl.
Inform.
0 (Vanilla)
0.246
0.204
0.703
0.150
0.096
0.547
134.786
Local VAE
a' 1e-3
0.491
0.407
0.814
0.390
0.305
0.686
107.636
le-2 2
0.449
0.385
0.797
0.173
0.132
0.635
123.288
a' = 1c- 1
0.457
0,432
0.626
0.945
0.796
0.996
49.364
0.424
0.406
0,594
0.977
0.800
0.999
43.194
lel
0.393
0,370
0,587
0.871
0.733
0.998
59.555
3 2
0.367
0.292
0.776
0.222
0.215
0.630
96.279
8-VAE
0.588
0.499
0.906
0.384
0.337
0.817
96.612
3 8
0.636
0.584
0.967
0.601
0.547
0.936
86.856
0.649
0.580
0.941
0.690
0.473
0.883
86.237
5.2 CONCATENATED DATASET OF THE CARS3D AND SMALLNORB
Finally, we evaluated proposed model the dataset, which explicitly has the locality. In this section, we concatenated two datasets: the Cars3D Dataset (Reed et al. (014) and the SmalINORB Dataset (LeCun al. 2004). These datasets are both of images of 3D objects. Each dataset has three (elevation, azimuth, and object type) and four (elevation, azimuth, category, and lighting condition) disentanglement factors. respectively. The elevation and azimuth are the global control factors common to the entire dataset, and the others are the sub-dataset specific factors. Each image has no information about which sub-dataset comes from. We refer to this dataset as the CarsNORB Dataset in the following. Note that learning this entire dataset is more challenging than learning each sub-dataset respectively since these two sub-datasets have different structures.
Table 2
shows the empirical evaluation on the CarsNORB Dataset. The hyperparameters follow the same setting as the experiment on the 3D Shapes Dataset. We calculated the DCI Disentanglement score for each sub-dataset. According to the table, the Disentanglement score of the Vanilla VAE was remarkably low for the Cars3D Dataset. We believe this result comes from the difference in statistics that the Cars3D Dataset (N = 17, 568) has fewer samples than the SmalINORB Dataset (N = 48, 600) and has a more complicated structure, including colors. The Disentanglement scores took maximum at 1c-2 for both sub-datasets. This result indicates that the locality enables the model to achieve the disentangled representation for each subspace without any label information.
Table 2: Quantitative evaluations of the Local VAE on the CarsNORB Dataset.
a (Vanilla)
a 1e-2
le-1
NORB Disentanglement
0.265
0.282 0.165
0.264
0.255
Cars Disentanglement
0.079
0.111
0.080
6 CONCLUSION
In this study, we proposed the Local VAE, a deep generative model suitable for datasets with local structure. Since conventional local learning approaches learn the embeddings at each neighborhood from scratch, integrating these approaches with deep neural networks, which require a massive amount of data and extended training time, was not reasonable. To overcome this trade-off, we performed gradient-based meta-learning, called MAML, with the supervision of past experiences outside the neighborhood. We evaluated our proposed model with the 3D Shapes dataset and the the concatenated dataset of the Cars3D and SmalINORB, which are one of the most straightforward datasets comprising disentangled local structures. Our experimental results showed that the learned representations of the Local VAE were more disentangled than that of the Vanilla VAE in terms of DCI scores. Moreover, the Local VAE improved the quality of the generated images compared with the Vanilla VAE according subjective evaluation and FID scores.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille, Tom Eccles, Loic Matthey, Chris Burgess, Nicholas Watters, Alexander Lerch- ner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems 31, pp. 9873-9883. 2018.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffmann, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6-8. Univ. of Texas, 1992.
Matthew Brand. Charting a manifold. In Advances in neural information processing systems, pp- 985-992, 2003.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://giiinbbcoo///pppmm/////ss 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-vae. In Workshop on Learning Disen- tangled Representations at the 31st Conference on Neural Information Processing Systems, 2017.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-cume con-curated data. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances Neural Information Processing Systems, pp- 2610-2620, 2018.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomczak. Hyper- spherical variational auto-encoders. 34th Conference Uncertainty in Artificial Intelligence, 2018.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
Paul Ekman and Dacher Keltner. Universal facial expressions of emotion. Segerstrale U, P. Molnar P. eds. Nonverbal communication: Where nature meets culture, pp. 27-46, 1997.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1126-1135. JMLR.org, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 2672-26622 2680, 2014.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference on Learning Representa- tions, 2018
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626 666366663.255 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-vae: Learning basic visual concepts with a con- strained variational framework. In Proceedings of the 6th International Conference on Learning Representations, 2017.

---

Under review as a conference paper at ICLR 2020
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In Interna- tional Conference on Learning Representations, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of the 5th International Conference on Learning Representations. 2017.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv 2017.
preprint
Nandakishore Kambhatla and Todd Leen. Dimension reduction by local principal component analysis. Neural Computation, 9(7):199--551,, 1997. doi: 10. 10.1162/neco. (tt62//ecc..99..9.. 1997.9.7.1499.
Charles Kemp and Joshua B. Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31):10687-10692, 2008. ISSN 0027-8424. doi: 10.1073/ mmas.0802631105.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning. pp. 2654-2663, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations, 2013.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 15006266):133221333333 2015.
Yann LeCun, Fu Jie Huang, Leon Bottou, et al. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004, pp. 97-104. Citeseer, 2004.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning. volume 97, pp. 41 14-4124. PMLR, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov) 2008.
Emile Mathieu, Charline Le Lan, Chris J. Maddison, Ryota Tomioka, and Yee Whye Teh. Hierar- chical representations with poincaré variational auto-encoders. CoRR, 0ss/19901.6033, 2019.
L. McInnes, J. Healy, and J. Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. CoRR, 2018
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsuper- vised learning rules. In International Conference Learning Representations, 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv: 17844444,, 2014.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped nor- mal distribution on hyperbolic space for gradient-based learning. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 4693-4702. PMLR, 2019.
Ivan Ovinnikov. Poincaré wasserstein autoencoder. CoRR,
2019.
10

---

Under review as a conference paper at ICLR 2020
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In Proceedings of the Sth International Conference on Learning Representations, 2017.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning to disentangle factors of variation with manifold interaction. In Erie P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research. pp. 1431-1439 2014.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learn- ing, volume 37 Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Con- ference on International Conference on Machine Learning Volume 32, pp. 1278-1286, 2014.
Jason Tyler Rolfe. Discrete variational autoencoders. In Proceedings of the 5th International Con- ference on Learning Representations, 2017.
Sam T. Roweis and Lawrence K. Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 29005500))23333-2356, 2000. ISSN 0036-8075. doi: 10.1 (..116/cceecce2200555555 2323.
Andrew M. Saxe, James I.. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 (237:11537-11546, 2019, ISSN 0027-8424. doi: 00073//pas..120226111
Jurgen Schmidhuber: Evolutionary principles self-referential learning. on learning now to learn: The meta-metta-meta..-hott 14 May 1987.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 3483-3491. Curran Associates, Inc., 2015.
Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500);2319 2223,,,2233222222 2000. ISSN 0036-8075. doi: 10.1 00.11266cciemee2225500...33
Pascal Vincent and Yoshua Bengio. Manifold parzen windows. In Advances in neural information processing systems, pp. 449-856, 2003.

---

Under review as a conference paper at ICLR 2020
A RELATIONSHIP BETWEEN THE LOCAL VAE AND A CONVENTIONAL LOCAL LEARNING
In this section, discuss that the Local VAE be interpreted the variant of the conventional local learning approaches. We introduce Local Linear Embedding (LL.) (Roweis & Saul 2000) as typical local learning algorithm. The LLE extracts low-dimensional brighborhood-preservvii em- beddings based on the precomputed neighbor graph. This method assumes that the dataset consists of combination of locally linear spaces, and applies linear projection each neighborhood. For the dataset D (x=") Ji=1 the objective function of LLE is defined
L(W) Σ x
Σ W Wayaa())
where parameter W is an N x N matrix. The element Wip of W is nonzero only when af(i belongs to the set of neighbors of (+), and E, Wij O. The neighbor graph of x(() is built by using the k-nearest neighbor method. Since Equation gis known not to have local minima, we can derive the solution of Equation 9by basic matrix calculation. Once the model parameter W is derived, we can obtain the low-dimensional embeddings 2(1), x(2), :.... ( ((N) of each data by minimizing the loss Σ. ||x(0) E, Wiyz(1) with respect z.
Here, we consider extending the embedding model of LLE from linear projection gen- eral nonlinear model. The embedding model of (+) corresponds to E, Wayzo()) in Equation 9 In other words, if we denote the index of the (s))s neighborhood as fi,-.., the model pa- rameters of the neighborhood are [Wij Wejar-..; Wajk]- In the following. generalize these [W W jjz: .... Wij parameter ( for the of neighborhoods N(x(1). Then, the aforementioned objective function is given by the following:
((ON( . (()(N))) = Σ (1)
90 ()) (N(a
(10)
Unlike Equation 9 there are no restrictions on the number of parameters or formulation, so opti- mization of the above equation is generally challenging. Notably, in the case of go xc..)) (.) being deep neural network, a massive amount of data and extended training time are required for each i-th neighborhood (a)))))
We can interpret Equation 10 as the general formulation the Local VAE with the Gaussian De- coder. Consider taking only (4) itself instead of set of neighborhoods ((x(()) of ax()) as input to the function ge (.) (-) in Equation 10 If we take the model parameters as 0 (2et)) and (
and use Autoencoder for the model g(-). Equation 10 corresponds to the objective function of the
Local VAE with the Gaussian Decoder.
B EXPERIMENTAL CONDITIONS AND HYPERPARAMETERS
In this section, we show the experimental conditions and hyperparameters which used for all the numerical experiments in the main text. (Table 3) shows the Encoder and the Decoder architectures of the VAE. We used the multivariate isotropic Gaussian for the latent variable. The outputs of the Encoder correspond to and log o of the variational poseterior distribution <(=|z)) Table 4 shows the hyperparameters for the model and the training procedure. In addition the parameters shown in the table, we used the gradient boosted trees from Scikit-learn with the default setting for computing the DCI scores. We also used the Inception-v3 network from Keras, which pre-trained on the ImageNet dataset to compute the FID.
12

---

Under review as a conference paper at ICLR 2020
Table 3: Network architecture for the numerical experiments.
Encoder
Decoder
Input: 64 x 64 X 3
Input: 11
4 X 4 conv, 32 ReLU, stride 2
FC, 256 ReLU
4 x 4 conv, 32 ReLU, stride 2
FC, 4 x x 64 ReLU
4 X 4 conv, 64 ReLU, stride 2
4 upconv, 64 ReLU, stride 2
4 X 4 conv, 64 ReLU, stride 2
X 4 upconv, 32 ReLU, stride
FC 256, F2 10
4 upconv, 32 ReLU, stride
x 4 upconv, 3, stride
Table 4: The hyperparameters which are used for the numerical experiments.
Parameter
Value
Batch size (corresponds to the number of tasks)
25
Inner batch size (corresponds K)
10
Latent space dimension
10
Optimizer
Adam
Adam: betal
0.9
Adam: beta2
0.999
Adam: epsilon
1e-8
Adam: learning rate
le-4
Decoder type
Bernoulli
Training steps
300,000
C LATENT INTERPOLATION
Figure 4: Interpolation of Local VAE's latent space. Each i-th corresponds to the reconstructed
image with the latent code 2; modified in the range of [-2,
Figure 4 shows the learned latent space of the Local VAE model. We swept each latent dimension for the specific training sample in the range of [- 2]. The model extracted the angle, shape, and size of the object as the disentangled factors. The color of the reconstructed images was not changed against the latent space interpolation. We believe that this because the model extracted the color information as global feature.
D COMPARISON OF K-NEIGHBOR CONSTRUCTION METHODS
As mentioned in the main text, we can arbitrarily choose the distance for neighborhood construction in Local VAE. We evaluated two types of methods: synthetic neighhorhood by sampling and k-
13

---

Under review as a conference paper at ICLR 2020
nearest neighborhood on latent space. Among them, although the synthetic neighborhood has the advantage that it is simple to implement and fast to compute, e distance on input space widely used in general. In this section, compare the performance of the synthetic neighborhood to the 2² distance on input space.
shows the comparison of the methods to compute neighborhood. We used the 3D Shapes Dataset and set the hyperparameter as a = 1. Both methods outperformed Vanilla VAE, and they achieved comparable scores with each other. We note that the synthetic neighborhood by sampling achieved slightly better performance than the L² distance on input space in this experiment. This result suggests that there is an appropriate distance for each dataset, which is not necessarily the la distance on input space. Choosing the appropriate distance will improve the quality of the learned representations and generated images practical use.
Table 5
Table 5: Quantitative comparison of the neighborhood construction on the 3D Shapes dataset.
DCI w/ Color
DCI w/o Color
FID
Disent.
Compl.
Inform.
Disent.
Compl.
Inform.
Sampling
0.424
0.406
0.594
0.977
0,800
0.999
43.194
12 on input space
0.397
0.342
0.582
0.676
0.520
0.813
49.187
Vanilla VAE
0.246
0.204
0.703
0.150
0.096
0.547
134.786
14


Reviews:
[{"review_id": "SJl0wftOtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a local VAE based on the model-agnostic meta-learning concept. Images generated based local VAE are shown to be better than those generated bu \\beta-VAE in general in terms of generation quality and disentanglement/compactness/informativeness.\n\nThis paper shows an interesting idea of viewing neighbourhood of a data point as a task to adopt the meta-learning concept to  train VAE which gives superior performance.\n\nHowever, this paper falls short regarding its overall organisation of the paper and the way that different concepts being articulated which make it hard to follow and read. It is not easy to understand the linkage between a task and a neighbourhood being advocated without re-reading the paper for a few times. It seems to me that very substantial effort is needed for the revision to reach the ICLR standard.\n\nThe introduction section should be carefully rewritten to explain clearly the notions of locality and structural relationship. As far as I can see in the subsequent sections, the locality/structural relationship seems to be referring to the neighbourhood either in the input space or the latent space. Also, the way of discussing local learning should be elaborated a bit to make it clear what exact notion of local learning  is being presented.\n\n\nSpecific comments:\n\nPage 2: \n“since the most dataset tends to be governed by the consistent rules of the physical world”\n->\n“since most dataset tends to be governed by the consistent rules of the physical world”\n\n“Such kind of dataset has a multi-scale structure from a local to a global scale.” - some grammatical issue\n\nPage 5:\nLine 4: Equation 3 seems not related to meta-learning\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HklJK-wRKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Incorporating the local learning approaches into the training of the deep generative models can potentially create a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments. However, the local learning approach is limited to using only a simple model, because complex models require a large amount of data and extended training time. This paper overcomes this trade-off based on the insight that the real-world dataset often shares some structural similarities between each neighborhood.\n\nPros:\nThe paper is well-written. It is easy for the reader to understand. The derivations in the paper are correct.\n\n\nThe motivation is plausible. It is reasonable to expect that each local subspace of the dataset shares some structure since most dataset tends to be governed by the consistent rules of the real world.\n\n\nThe numerical experiments show that the locality enables the model to achieve the disentangled representation for each subspace without any label information.\n\nCons:\nThe novelty seems a little straight-forward. The paper just extends the objective function of the VAE to have different parameters for each local subset.\n\nIn consideration of extended training time in the complex model, the paper doesn't provide an evaluation of the efficiency of their proposed model.\n\nThe paper isn't very polished yet. There were more than a few spelling and grammatical errors, please proofread the work and improve the writing.\n\nThis paper in its current form is already fairly good. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bygxx5IpFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors of this paper propose to utilize the embedding model for the other local structure as a weak form of supervision based on the insight that the real-world datasets often shares some structural similarity between each neighborhood. The Local VAE is proposed to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning.\n\nLocal VAE incorporates local information by using prior distributions of local parameters in VAE. The overall model performs probabilistic inference via the conditional distribution from the meta parameters. There are several concerns:\n1. In section 2, authors discussed LLE. It is unclear the purpose of the section 2.1? \n2. LLE does not require W is nonnegative only, and \\sum_j W_{i,j}=0 is also contradictory with the nonnegative assumption. \n3. Authors claimed that Local VAE algorithm corresponds to the assumption that the dataset approximately lies on multiple subsets and each subset is generated from different parameters. It is unclear what is the connection of the Local VAE to multi-scale structures of the datasets.\n4.  Authors evaluated neighbors by sampling and k-nearest neighbors on latent space. It is unclear why not use the common k-nearest neighbors on the input data. K-nearest search should not be a computational problem for large datasets by using fast approach.\n\nAs the motivations of this paper, existing methods require massive amount of data and extended training time. However, authors did not demonstrate these points by comparing the proposed method with existing methods.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a structured VAE, where the model parameters depend on a local structure (such as distance in feature or local space), and it uses the meta-learning framework to adjust the dependency of the model parameters to the local neighborhood.

The idea is natural, as pointed by Rev#1. It incurs an extra learning cost, as noted by Rev#1 and #2, asking for details about the extra-cost. The authors' reply is (last alinea in first reply to Rev#1): we did not comment (...) because in essence, using neighborhoods in a naive way is not affordable. 
The area chair would like to know the actual computational time of Local VAE compared to that of the baselines.  

More details (for instance visualization) about the results on Cars3D and NORB would also be needed to better appreciate the impact of the locality structure. The fact that the optimal value (wrt Disentanglement) is rather low ($10^{-2}$) would need be discussed, and assessed w.r.t. the standard deviation.  

In summary, the paper presents a good idea. More details about its impacts on the VAE quality, and its computation costs, are needed to fully appreciate its merits. 
Decision: Reject