Year: 2019

Abstract:
Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.

text: Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning. The real-world datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood, respectively. However, this approach is only limited to use with a simple model, since the complex model, including deep neural networks, requires a massive amount of data and extended training time. In this study, we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood. We propose to utilize the embedding model for the other local structure as a weak form of supervision. Our proposed model, the Local VAE, generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning. Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure, including the 3D Shapes Dataset, and generated high-quality images.

heading: 1 INTRODUCTION
text: Extracting the hidden structure of the external environment is essential for achieving intelligent agents and modeling human learning (Kemp & Tenenbaum, 2008; Lake et al., 2015; Higgins et al., 2017; Achille et al., 2018; Saxe et al., 2019). Human beings and/or animals can effectively learn internal representations from a few amounts of experiences. Various methods of nonlinear feature extraction (Maaten & Hinton, 2008; McInnes et al., 2018) are recently proposed to model the complex environment. In addition, thanks to the developments of deep generative models (Kingma & Welling, 2013; Rezende et al., 2014; Goodfellow et al., 2014; Rezende & Mohamed, 2015), we can now handle the high-dimensional dataset on many individual problems.
Although recent studies succeeded in modeling the dataset for the specific problems, there are still challenging properties in real-world. The datasets that we are interested in are often characterized by the locality: the change in the structural relationship between the data points depending on location in observation space. For instance, a sequence of experiences gradually changes according to multiple aspects, including time, space, and modality; we need to identify each individual during the development of their faces consistently. Besides, the human-made objects often have multiple color options for the same shape or size. Many studies have incorporated locality for dimensionality reduction and representational learning (Kambhatla & Leen, 1997; Roweis & Saul, 2000; Tenenbaum et al., 2000). For example, combining the local learning approach with classical unsupervised learning algorithms such as PCA significantly improves their model capacity. These studies aim to find mappings between the data and the coordinate space under the assumption that the data space is composed of multiple low-dimensional subspaces (Brand, 2003; Vincent & Bengio, 2003). We refer this approach to as local learning. They usually use a linear projection for embedding models and a `2 distance in the input space for a neighborhood construction. In general, we can arbitrarily choose the distance for the neighbor graph, and it affects the quality of the embeddings.
Incorporating the local learning approaches into the training of the deep generative models will give us a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments at the same time. However, the integration of these two paradigms is not trivial. The conventional local learning approaches train the different embedding model for each neighborhood from scratch, nevertheless the deep neural networks generally require a large amount of data and take a long training time (LeCun et al., 2015). In other words, local learning approaches learn internal representations for each neighborhood by using only relatively simple models, whereas deep generative models learn one complex representation as a whole with deep neural networks.
To overcome this trade-off, the structural similarity between each neighborhood is the key. In the case of the human face, although each face varies greatly depending on age, gender, and etc., there are common facial expressions (Ekman & Keltner, 1997). It is reasonable to expect that each local subspace of the dataset shares some structure since most dataset tends to be governed by the consistent rules of the physical world (Achille et al., 2018). Under the assumptions about the locality and structural similarity, a dataset has two scales of structures: the local structure inside some neighborhood and the global relationship between each neighborhood. Figure 1a visualizes these two scales inside the dataset.
When a dataset has structural similarity, meta-learning is an effective approach. Meta-learning is an algorithm to learn the rules for each task quickly for a dataset consisting of multiple tasks (Schmidhuber, 1987; Bengio et al., 1992; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017). In this study, by considering each local neighborhood as a task for meta-learning, we extract the transferable knowledge between each local structure. We propose to train the meta embedding model, which parameters capture the common local structure and quickly adapt to each subspace by utilizing the structural similarity.
We generalize the typical deep generative model called the Variational Autoencoder (VAE) (Kingma & Welling, 2013; Rezende et al., 2014) to be applicable to the dataset with local structure. We extend the graphical model of the VAE to have different model parameters for each local subset of the dataset (Figure 1b) while keeping to avoid a large amount of computation by using the gradientbased meta-learning (Finn et al., 2017; Grant et al., 2018). By treating the neighborhood of each data point as a task to adopt the meta-learning, we make our proposed Local VAE possible to learn similar structures between neighbors quickly. We evaluate the performance of our proposed model with the 3D Shapes Dataset (Burgess & Kim, 2018) and the concatenated dataset of the Cars3D (Reed et al., 2014) and SmallNORB (LeCun et al., 2004). The numerical experiments shows that the locality enables the model to achieve the disentangled representation for each subspace without any label information.

heading: 2 BACKGROUND

heading: 2.1 VARIATIONAL AUTOENCODER
text: First, we introduce the VAE (Kingma & Welling, 2013; Rezende et al., 2014), which is one of the deep generative models that have been studied extensively in recent years. The objective function
of the VAE is defined as the variational lower bound of the log-likelihood (referred as the evidence lower bound, ELBO) for the dataset. Given dataset D = {x(i)}Ni=1, ELBO is defined as follows for each x(i);
log pθ(x (i)) ≥ Eqφ(z|x(i)) [ log pθ(x (i)|z) ] −DKL ( qφ(z|x(i)) ∥∥∥∥pθ(z)) = −L(θ,φ;x(i)), (1) where pθ(x(i)|z) is the conditional likelihood referred to as the decoder, and qφ(z|x(i)) is the variational posterior distribution referred to as the encoder. The choice of the prior pθ is typically the standard normal, and the posterior distribution is also variationally approximated by a Gaussian. This parametric formulation of qφ is called the reparameterization trick and enables the evaluation of the gradient of the objective function with respect to the network parameters. Overall, we can train the decoder and the encoder networks by taking the minimum of the negative ELBO using the gradient descent method.

heading: 2.2 MODEL-AGNOSTIC META-LEARNING
text: Then, to incorporate the local learning approach into the VAE, we utilize the Model-Agnostic MetaLearning (MAML) (Finn et al., 2017), which is a gradient-based meta-learning algorithm. The goal of MAML is to find task-independent knowledge from a number of previous related tasks. Once the meta-learner learns the task-independent knowledge, it can quickly adapt to a new task using only a few data points and training iterations. For connection to the deep generative models, we introduce the setting based on the maximum likelihood estimation described in Grant et al. (2018) instead of the original MAML formulation. In the setting of MAML, each data point is assumed to be sampled from the task-specific distribution x(i1), . . . ,x(iK) ∼ pTi(x). The MAML objective function in a maximum likelihood setting is
L(θ) = 1 N ∑ i
[ 1
K ∑ m − log p ( x(iK+m) | θ − α∇θ 1 K ∑ n
− log p(x(in)|θ)︸ ︷︷ ︸ θ′Ti
)] , (2)
where θ′Ti is the task-specific parameters after a single batch update by gradient descent from θ. The meta-learner can achieve the parameter θ, which can quickly adapt to new tasks with a small amount of data by optimizing Equation 2 using the gradient method. We note that θ can be interpreted as the parameters of the prior distribution for the task-specific parameters θTi . By replacing the expectation w.r.t. the original posterior distribution by the maximum likelihood estimate∫ f(θT )p(θT |θ)dθT ' f(θ′T ), the abovementioned objective function (Equation 2) recovers.

heading: 3 LOCAL VARIATIONAL AUTOENCODER
text: In this section, we will present the Local VAE, a variant of the VAE suitable for representation learning of a dataset with local structure.
Here, we extend the objective function of the VAE (Equation 1) to have different parameters for each local subset. We consider the variational lower bound of the log-likelihood for the dataset D, just as with the Vanilla VAE. However, we define the different model parameters θN(x(i)) and φN(x(i)) for each neighborhood N(x(i)) of the i-th data, respectively. Since these parameters are often highdimensional and require a long time and a large amount of data for training, we give meta parameters θ and φ as prior distributions of these local parameters. The overall model performs the probabilistic inference through the conditional distribution from the meta parameters. The variational lower
bound for the log-likelihood can be calculated as follows:
log pθ(x (i)) = log ∫ p(x(i)|z,θN(x(i)))p(z)p(θN(x(i))|θ)dzdθN(x(i)) (3)
≥ ∫ q(z|x(i),φN(x(i)))q(φN(x(i))|φ)×
log p(x(i)|z,θN(x(i)))p(z)p(θN(x(i))|θ) q(z|x(i),φN(x(i)))q(φN(x(i))|φ) dzdθN(x(i))dφN(x(i)) (4)
=Eq(z|x(i),φ N(x(i)) )q(φ N(x(i)) |φ)
[ log p(x(i)|z,θN(x(i)))p(θN(x(i))|θ) ] −DKL ( q(z|x(i),φN(x(i)))q(φN(x(i))|φ)‖p(z) ) , (5)
where p(θN(x(i))|θ) and q(φN(x(i))|φ) are the conditional distribution of the local parameters. We note that the integral variables of the expectation and the Kullback-Leibler divergence in Equation 5 are z, θN(x(i)) and φN(x(i)).
As we mentioned above, the integral variables of Equation 5 include θN(x(i)) and φN(x(i)). This means that Equation 5 needs to take an integral of the model parameters to evaluate the objective function, while the one of Vanilla VAE only requires the Monte Carlo expectation of z. Such an integral is unreasonable in deep generative models where model parameters are often high-dimensional. To overcome this problem, we replace this integral with the maximum likelihood estimator updated by the one-step gradient method, as we described in Section 2.2. Let L(θ,φ;x(i)) be the negative of the expression obtained by Equation 5. By replacing the integral of θN(x(i)) and φN(x(i)) with the maximum likelihood estimator θ′N(x(i)) and φ ′ N(x(i)), we obtain
L(θ,φ;x(i)) '− Eq(z|x(i),φ′ N(x(i)) )
[ log p(x(i)|z,θ′N(x(i))) ] +DKL ( q(z|x(i),φ′N(x(i))) ∥∥∥∥pθ(z)) =Lg(θ′N(x(i)),φ ′ N(x(i));x (i)). (6)
We note that the integral variable of Equation 6 is now only z. The maximum likelihood estimator of the local parameters can be obtained by the following update rule:
θ′N(x(i)) =θ − α∇θ 1
K ∑ x∈N(x(i)) L(θ,φ;x), (7)
φ′N(x(i)) =φ− α∇φ 1
K ∑ x∈N(x(i)) L(θ,φ;x), (8)
where K is the number of neighborhoods for x(i). L(θ,φ;x) in the above equations is the ELBO of Vanilla VAE defined in Equation 1. Algorithm 1 shows the overall algorithm.
From the perspective of the graphical model, our proposed Local VAE algorithm corresponds to the assumption that the dataset approximately lies on multiple subsets and that each subset is generated from different parameters. Alternatively, from the viewpoint of meta-learning, our objective function is consistent with the case of training VAE by MAML when task information is given as a neighbor graph. We can also give the relationship of our model to the conventional local learning approach. Please see Appendix A for the detail.

heading: 3.1 NEIGHBORHOOD CONSTRUCTION
text: As we mentioned above, local learning approaches have to construct a neighbor graph before training the model. The conventional approaches often use the k-nearest neighbor graph build on the original data space. In general, we can make arbitrarily choice how to construct the neighborhood, and it affects the quality of the embeddings. We evaluated two types of neighborhood in the following experiments: synthetic neighborhood by sampling and k-nearest neighborhood on latent space. In the synthetic neighborhood by sampling, we sampled K different examples for each x(i) from the noise distribution assumed as the observation process of the data and used these examples as
Algorithm 1 Optimization of Local VAEs 1: while until converge do 2: for x(i) in mini-batch B do 3: Sample K-points from the neighborhood of x(i): x(1), . . . ,x(K) ∼ N(x(i)). 4: Evaluate the local objective L(θ,φ;x) for the K-neighborhood w.r.t. the meta param-
eters based on Equation 1. 5: Update the local parameters: θN(x(i)) ← θ − α∇θ 1K ∑ x∈N(x(i)) L(θ,φ;x),
φN(x(i)) ← φ− α∇φ 1K ∑ x∈N(x(i)) L(θ,φ;x).
6: Evaluate the global objective Lg(θN(x(i)),φN(x(i));x(i)) for i-th data w.r.t. the local parameters based on Equation 6. 7: end for 8: Update the meta parameters:
θ ← θ − η∇θ 1|B| ∑
i∈B Lg(θN(x(i)),φN(x(i));x(i)), φ← φ− η∇φ 1|B| ∑ i∈B Lg(θN(x(i)),φN(x(i));x(i)).
9: end while
the neighborhood of x(i). We considered that this method is effective when the data is densely distributed in the observation space and used this method for 3D Shapes Dataset to omitting the time to construct the neighbor graph for the large dataset. On the other hand, in the k-nearest neighborhood on latent space, we used the k-nearest neighbor graph builds on the latent space of the VAE. We can expect that we can obtain a neighborhood that follows our intuition by using the distance on latent space rather than the input space (Caron et al., 2019). In the experiment on the CarsNORB Dataset, which we will describe later, we used Faiss (Johnson et al., 2017) for similarity search and continuously updated the latent code for each iteration during the training phase.

heading: 4 RELATED WORKS
text: In this study, we employed the gradient-based meta-learning method MAML (Finn et al., 2017) and its probabilistic formulation (Grant et al., 2018) to find local parameters from a few data points. Recently, several studies (Hsu et al., 2019; Metz et al., 2019) proposed the integration of unsupervised learning and meta-learning from another perspective. Hsu et al. (2019) proposed the algorithm for generating MAML task information by utilizing embedded similarity information created with unsupervised learning. In contrast to this case of using unsupervised learning for meta-learning, we used meta-learning to perform unsupervised learning. In addition, Metz et al. (2019) proposed a way to seek the objective function itself for representation learning with meta-learning.
The local learning approaches, including LLE (Roweis & Saul, 2000) and Isomap (Tenenbaum et al., 2000), are deeply related to our work. We discuss the relationship between LLE and our work in Appendix A in detail. Besides, the extension of generative models to make them applicable for structured datasets has recently been extensively studied. The generalization of the latent space of the VAE to a non-Euclidean space such as a spherical surface (Davidson et al., 2018), hyperbolic space (Ovinnikov, 2019; Nagano et al., 2019; Mathieu et al., 2019), or discrete space (Jang et al., 2017; Rolfe, 2017) was proposed.
The property of disentanglement has attracted notable attention in structure extraction using VAEs as described above (Higgins et al., 2017; Burgess et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018; Locatello et al., 2019). Most of the proposed models try to realize disentanglement representation by modifying the penalty term of the objective function or network architectures. On the other hand, our approach focuses on how to learn parameters suitable for disentangled (local) representations so that we can utilize both these aforementioned techniques and our proposed method at the same time.
From the viewpoint of generating data by a deep generative model with some supervision, conditional generation is commonly practiced (Kingma et al., 2014; Sohn et al., 2015; Mirza & Osindero, 2014). Our method is similar to these approaches in that the density function is conditioned on the
neighborhood of the specific data point. While conventional conditional generation generates data by only one model parameter with the known class label as an additional latent code, our proposed model has different network parameters for each neighborhood. Moreover, our approach is more applicable than conventional methods since our approach does not need any class label information.

heading: 5 NUMERICAL EVALUATIONS

heading: 5.1 3D SHAPES DATASET
text: Here, we numerically evaluate the performance of the Local VAE. We use the 3D Shapes Dataset (Burgess & Kim, 2018), which has a clear disentangled property. The disentangled property can be interpreted as the simplest case of the local structure. The disentangled dataset is assumed to be able to control by a small number of factors. Since these factors alter the observation in data space, and the scale of them is different one by one, we can interpret the factors which significantly affect the observation as the global features and other factors as the local features.
We followed all the experimental settings in Locatello et al. (2019), except the batch size and the number of tasks, to eliminate effects outside the proposed method as much as possible. Please see Appendix B for the detail. We qualitatively assess the generated images and quantitatively evaluate the model performance by using the disentanglement metric (DCI scores) proposed by Eastwood & Williams (2018) and the Fréchet Inception Distance (FID) (Heusel et al., 2017).
Figure 2 shows the conditionally generated images of the trained models. At the inference phase, the model obtains the local parameters θN(x(i)) by applying one-step gradient descent using the randomly selected training data and generates images from these local parameters. We trained multiple models with different values of α, which is the hyperparameter of Local VAEs. Note that the original objective function of Vanilla VAEs recovers in the case of α = 0 since the local parameters are strictly consistent with the meta parameters. According to the subjective assessment, the quality of generated images is better when α is large.
There could be a concern that overfitting caused the result above. If the model obtains local parameters that perfectly generate only the training sample to be referenced, the quality of the generated image will be superficially high. To exclude this possibility, we visualized the reference samples and their corresponding generated images of the model with α = 1 in Figure 3. The leftmost column shows the reference training samples used for the conditional localized generations. Each row visualizes the generated images conditioned on the reference sample in the left column. We randomly picked ten latent codes z1, . . . ,z10 from the prior distribution, and then used these codes for every conditional generation. In other words, the images shown in the same column share their latent code. According to the figure, the trained model generated clearly different images in the same row conditioned on one training sample. This result strongly suggests that the Local VAE model did not overfit to the specific data. Moreover, the shape, angle, and size of the object were the same, and only the color was different in each column. These results suggest that the model trained by the proposed method segregated color information as global features and other information as local
= TKNVVN L/
features, and obtained an internal representation independent of the global features. Note that Local VAEs only use neighborhood relationships and do not use any label information.
Then, we qualitatively evaluated the latent representations of Local VAEs by using the DCI scores (Eastwood & Williams, 2018). The DCI scores quantify the learned representations based on three types of aspects: Disentanglement, Compactness, and Informativeness. All metrics can be computed from the importance of each dimension of the latent space for predicting a factor of variation. DCI scores require the label information of the ground truth. Since the Local VAE clearly extracted the color information as the global feature, we calculated DCI scores for the two types of conditions: the class labels including all six aspects (w/ Color condition), and the class labels excluding color information (w/o Color condition).
Table 1 shows the empirical evaluations of the DCI scores. DCI scores with six labels (w/ Color condition) of the Local VAE were slightly better than the one for the Vanilla VAE. The model with small α, which is the value closer to the Vanilla VAE, tended to achieve better scores for all DCI metrics in the Local VAE comparison. This result is attributed to the loss of color information from the internal representation as α increases. On the other hand, the Local VAEs significantly improved the DCI scores in the condition without color. All DCI metrics took their maximum value at α = 1. The performance was slightly degraded at α = 1e1, and the loss diverged during training at α = 1e2. The numerical evaluation suggests that α can control how much of the structure behind the entire dataset is considered as global variation and from where it is regarded as a local variation. We also evaluated the performance of the β-VAE (Higgins et al., 2017) as a reference. The β-VAE modifies the KL term (Equation 1) by multiplying non-zero coefficient β. Although the β-VAE with β = 8 or β = 16 achieved higher scores than the Local VAE in the condition with color, the Local VAE with α = 1 significantly outperformed all the β-VAE in the condition without color.
We also evaluated the quality of the generated images with the Fréchet Inception Distance (FID) (Heusel et al., 2017). FID is a metric that evaluates the similarity of quality between real and generated images. We used the 50,000 samples of the ground truth dataset and generated images for FID calculation. According to Table 1, FID tended to be low at the large α and took the minimum value at α = 1. This result was consistent with the DCI scores of the condition without color. Since the method to calculate neighborhood is arbitrary, we also compared the synthetic neighborhood with a widely used approach. Please see Appendix D for the detail.

heading: 5.2 CONCATENATED DATASET OF THE CARS3D AND SMALLNORB
text: Finally, we evaluated our proposed model on the dataset, which explicitly has the locality. In this section, we concatenated two datasets: the Cars3D Dataset (Reed et al., 2014) and the SmallNORB Dataset (LeCun et al., 2004). These datasets are both set of images of 3D objects. Each dataset has three (elevation, azimuth, and object type) and four (elevation, azimuth, category, and lighting condition) disentanglement factors, respectively. The elevation and azimuth are the global control factors common to the entire dataset, and the others are the sub-dataset specific factors. Each image has no information about which sub-dataset it comes from. We refer to this dataset as the CarsNORB Dataset in the following. Note that learning this entire dataset is more challenging than learning each sub-dataset respectively since these two sub-datasets have different structures.
Table 2 shows the empirical evaluation on the CarsNORB Dataset. The hyperparameters follow the same setting as the experiment on the 3D Shapes Dataset. We calculated the DCI Disentanglement score for each sub-dataset. According to the table, the Disentanglement score of the Vanilla VAE was remarkably low for the Cars3D Dataset. We believe this result comes from the difference in statistics that the Cars3D Dataset (N = 17, 568) has fewer samples than the SmallNORB Dataset (N = 48, 600) and has a more complicated structure, including colors. The Disentanglement scores took maximum at α = 1e−2 for both sub-datasets. This result indicates that the locality enables the model to achieve the disentangled representation for each subspace without any label information.

heading: 6 CONCLUSION
text: In this study, we proposed the Local VAE, a deep generative model suitable for datasets with local structure. Since conventional local learning approaches learn the embeddings at each neighborhood from scratch, integrating these approaches with deep neural networks, which require a massive amount of data and extended training time, was not reasonable. To overcome this trade-off, we performed gradient-based meta-learning, called MAML, with the supervision of past experiences outside the neighborhood. We evaluated our proposed model with the 3D Shapes dataset and the the concatenated dataset of the Cars3D and SmallNORB, which are one of the most straightforward datasets comprising disentangled local structures. Our experimental results showed that the learned representations of the Local VAE were more disentangled than that of the Vanilla VAE in terms of DCI scores. Moreover, the Local VAE improved the quality of the generated images compared with the Vanilla VAE according to subjective evaluation and FID scores.

heading: A RELATIONSHIP BETWEEN THE LOCAL VAE AND A CONVENTIONAL LOCAL LEARNING
text: In this section, we discuss that the Local VAE can be interpreted as the variant of the conventional local learning approaches. We introduce Local Linear Embedding (LLE) (Roweis & Saul, 2000) as a typical local learning algorithm. The LLE extracts low-dimensional neighborhood-preserving embeddings based on the precomputed neighbor graph. This method assumes that the dataset consists of a combination of locally linear spaces, and applies a linear projection to each neighborhood. For the dataset D = {x(i)}Ni=1, the objective function of LLE is defined as
L(W ) = ∑ i ∥∥∥∥x(i) −∑ j Wijx (j) ∥∥∥∥2, (9) where parameter W is an N ×N matrix. The element Wij of W is nonzero only when x(j) belongs to the set of neighbors of x(i), and ∑ j Wij = 0. The neighbor graph of x
(i) is built by using the k-nearest neighbor method. Since Equation 9 is known not to have local minima, we can derive the solution of Equation 9 by basic matrix calculation. Once the model parameter W is derived, we can obtain the low-dimensional embeddings z(1), z(2), . . . ,z(N) of each data by minimizing the loss∑
i ‖z(i) − ∑ j Wijz (j)‖2 with respect to z.
Here, we consider extending the embedding model of LLE from a linear projection to a general nonlinear model. The embedding model of x(i) corresponds to ∑ j Wijx
(j) in Equation 9. In other words, if we denote the index of the x(i)’s neighborhood as j1, . . . , jK , the model parameters of the neighborhood are [Wij1 ,Wij2 , . . . ,WijK ]. In the following, we generalize these [Wij1 ,Wij2 , . . . ,WijK ] as parameter θN(x(i)) for the set of neighborhoods N(x
(i)). Then, the aforementioned objective function is given by the following:
L ( θN(x(1)), . . . ,θN(x(N)) ) = ∑ i ∥∥∥∥x(i) − gθN(x(i))(N(x(i))) ∥∥∥∥2. (10)
Unlike Equation 9, there are no restrictions on the number of parameters or formulation, so optimization of the above equation is generally challenging. Notably, in the case of gθ
N(x(i)) (·) being
a deep neural network, a massive amount of data and extended training time are required for each i-th neighborhood N(x(i)).
We can interpret Equation 10 as the general formulation of the Local VAE with the Gaussian Decoder. Consider taking only x(i) itself instead of a set of neighborhoods N(x(i)) of x(i) as input to the function gθ
N(x(i)) (·) in Equation 10. If we take the model parameters as θN(x(i)) and φN(x(i))
and use Autoencoder for the model g(·), Equation 10 corresponds to the objective function of the Local VAE with the Gaussian Decoder.

heading: B EXPERIMENTAL CONDITIONS AND HYPERPARAMETERS
text: In this section, we show the experimental conditions and hyperparameters which are used for all the numerical experiments in the main text. Table 3 shows the Encoder and the Decoder architectures of the VAE. We used the multivariate isotropic Gaussian for the latent variable. The outputs of the Encoder correspond to µ and logσ of the variational poseterior distribution q(z|x). Table 4 shows the hyperparameters for the model and the training procedure. In addition to the parameters shown in the table, we used the gradient boosted trees from Scikit-learn with the default setting for computing the DCI scores. We also used the Inception-v3 network from Keras, which is pre-trained on the ImageNet dataset to compute the FID.

heading: C LATENT INTERPOLATION
text: Figure 4 shows the learned latent space of the Local VAE model. We swept each latent dimension for the specific training sample in the range of [−2, 2]. The model extracted the angle, shape, and size of the object as the disentangled factors. The color of the reconstructed images was not changed against the latent space interpolation. We believe that this is because the model extracted the color information as a global feature.
D COMPARISON OF k-NEIGHBOR CONSTRUCTION METHODS
As mentioned in the main text, we can arbitrarily choose the distance for neighborhood construction in Local VAE. We evaluated two types of methods: synthetic neighborhood by sampling and k-
nearest neighborhood on latent space. Among them, although the synthetic neighborhood has the advantage that it is simple to implement and fast to compute, `2 distance on input space is widely used in general. In this section, we compare the performance of the synthetic neighborhood to the `2 distance on input space.
Table 5 shows the comparison of the methods to compute neighborhood. We used the 3D Shapes Dataset and set the hyperparameter as α = 1. Both methods outperformed Vanilla VAE, and they achieved comparable scores with each other. We note that the synthetic neighborhood by sampling achieved slightly better performance than the `2 distance on input space in this experiment. This result suggests that there is an appropriate distance for each dataset, which is not necessarily the `2 distance on input space. Choosing the appropriate distance will improve the quality of the learned representations and generated images in practical use.

Review summaries:
Incorporating the local learning approaches into the training of the deep generative models can potentially create a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments. However, the local learning approach is limited to using only a simple model, because complex models require a large amount of data and extended training time. This paper overcomes this trade-off based on the insight that the real-world dataset often shares some structural similarities between each neighborhood.