Localized Generations with Deep Neural Networks for Multi-Scale Structured Datasets

Extracting the hidden structure of the external environment is an essential component of intelligent agents and human learning . The real-world datasets that we are interested in are often characterized by the locality : the change in the structural relationship between the data points depending on location in observation space . The local learning approach extracts semantic representations for these datasets by training the embedding model from scratch for each local neighborhood , respectively . However , this approach is only limited to use with a simple model , since the complex model , including deep neural networks , requires a massive amount of data and extended training time . In this study , we overcome this trade-off based on the insight that the real-world dataset often shares some structural similarity between each neighborhood . We propose to utilize the embedding model for the other local structure as a weak form of supervision . Our proposed model , the Local VAE , generalize the Variational Autoencoder to have the different model parameters for each local subset and train these local parameters by the gradient-based meta-learning . Our experimental results showed that the Local VAE succeeded in learning the semantic representations for the dataset with local structure , including the 3D Shapes Dataset , and generated high-quality images . 1 INTRODUCTION . Extracting the hidden structure of the external environment is essential for achieving intelligent agents and modeling human learning ( Kemp & Tenenbaum , 2008 ; Lake et al. , 2015 ; Higgins et al. , 2017 ; Achille et al. , 2018 ; Saxe et al. , 2019 ) . Human beings and/or animals can effectively learn internal representations from a few amounts of experiences . Various methods of nonlinear feature extraction ( Maaten & Hinton , 2008 ; McInnes et al. , 2018 ) are recently proposed to model the complex environment . In addition , thanks to the developments of deep generative models ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ; Goodfellow et al. , 2014 ; Rezende & Mohamed , 2015 ) , we can now handle the high-dimensional dataset on many individual problems . Although recent studies succeeded in modeling the dataset for the specific problems , there are still challenging properties in real-world . The datasets that we are interested in are often characterized by the locality : the change in the structural relationship between the data points depending on location in observation space . For instance , a sequence of experiences gradually changes according to multiple aspects , including time , space , and modality ; we need to identify each individual during the development of their faces consistently . Besides , the human-made objects often have multiple color options for the same shape or size . Many studies have incorporated locality for dimensionality reduction and representational learning ( Kambhatla & Leen , 1997 ; Roweis & Saul , 2000 ; Tenenbaum et al. , 2000 ) . For example , combining the local learning approach with classical unsupervised learning algorithms such as PCA significantly improves their model capacity . These studies aim to find mappings between the data and the coordinate space under the assumption that the data space is composed of multiple low-dimensional subspaces ( Brand , 2003 ; Vincent & Bengio , 2003 ) . We refer this approach to as local learning . They usually use a linear projection for embedding models and a ` 2 distance in the input space for a neighborhood construction . In general , we can arbitrarily choose the distance for the neighbor graph , and it affects the quality of the embeddings . Incorporating the local learning approaches into the training of the deep generative models will give us a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments at the same time . However , the integration of these two paradigms is not trivial . The conventional local learning approaches train the different embedding model for each neighborhood from scratch , nevertheless the deep neural networks generally require a large amount of data and take a long training time ( LeCun et al. , 2015 ) . In other words , local learning approaches learn internal representations for each neighborhood by using only relatively simple models , whereas deep generative models learn one complex representation as a whole with deep neural networks . To overcome this trade-off , the structural similarity between each neighborhood is the key . In the case of the human face , although each face varies greatly depending on age , gender , and etc. , there are common facial expressions ( Ekman & Keltner , 1997 ) . It is reasonable to expect that each local subspace of the dataset shares some structure since most dataset tends to be governed by the consistent rules of the physical world ( Achille et al. , 2018 ) . Under the assumptions about the locality and structural similarity , a dataset has two scales of structures : the local structure inside some neighborhood and the global relationship between each neighborhood . Figure 1a visualizes these two scales inside the dataset . When a dataset has structural similarity , meta-learning is an effective approach . Meta-learning is an algorithm to learn the rules for each task quickly for a dataset consisting of multiple tasks ( Schmidhuber , 1987 ; Bengio et al. , 1992 ; Andrychowicz et al. , 2016 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ) . In this study , by considering each local neighborhood as a task for meta-learning , we extract the transferable knowledge between each local structure . We propose to train the meta embedding model , which parameters capture the common local structure and quickly adapt to each subspace by utilizing the structural similarity . We generalize the typical deep generative model called the Variational Autoencoder ( VAE ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) to be applicable to the dataset with local structure . We extend the graphical model of the VAE to have different model parameters for each local subset of the dataset ( Figure 1b ) while keeping to avoid a large amount of computation by using the gradientbased meta-learning ( Finn et al. , 2017 ; Grant et al. , 2018 ) . By treating the neighborhood of each data point as a task to adopt the meta-learning , we make our proposed Local VAE possible to learn similar structures between neighbors quickly . We evaluate the performance of our proposed model with the 3D Shapes Dataset ( Burgess & Kim , 2018 ) and the concatenated dataset of the Cars3D ( Reed et al. , 2014 ) and SmallNORB ( LeCun et al. , 2004 ) . The numerical experiments shows that the locality enables the model to achieve the disentangled representation for each subspace without any label information . 2 BACKGROUND . 2.1 VARIATIONAL AUTOENCODER . First , we introduce the VAE ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , which is one of the deep generative models that have been studied extensively in recent years . The objective function of the VAE is defined as the variational lower bound of the log-likelihood ( referred as the evidence lower bound , ELBO ) for the dataset . Given dataset D = { x ( i ) } Ni=1 , ELBO is defined as follows for each x ( i ) ; log pθ ( x ( i ) ) ≥ Eqφ ( z|x ( i ) ) [ log pθ ( x ( i ) |z ) ] −DKL ( qφ ( z|x ( i ) ) ∥∥∥∥pθ ( z ) ) = −L ( θ , φ ; x ( i ) ) , ( 1 ) where pθ ( x ( i ) |z ) is the conditional likelihood referred to as the decoder , and qφ ( z|x ( i ) ) is the variational posterior distribution referred to as the encoder . The choice of the prior pθ is typically the standard normal , and the posterior distribution is also variationally approximated by a Gaussian . This parametric formulation of qφ is called the reparameterization trick and enables the evaluation of the gradient of the objective function with respect to the network parameters . Overall , we can train the decoder and the encoder networks by taking the minimum of the negative ELBO using the gradient descent method . 2.2 MODEL-AGNOSTIC META-LEARNING . Then , to incorporate the local learning approach into the VAE , we utilize the Model-Agnostic MetaLearning ( MAML ) ( Finn et al. , 2017 ) , which is a gradient-based meta-learning algorithm . The goal of MAML is to find task-independent knowledge from a number of previous related tasks . Once the meta-learner learns the task-independent knowledge , it can quickly adapt to a new task using only a few data points and training iterations . For connection to the deep generative models , we introduce the setting based on the maximum likelihood estimation described in Grant et al . ( 2018 ) instead of the original MAML formulation . In the setting of MAML , each data point is assumed to be sampled from the task-specific distribution x ( i1 ) , . . . , x ( iK ) ∼ pTi ( x ) . The MAML objective function in a maximum likelihood setting is L ( θ ) = 1 N ∑ i [ 1 K ∑ m − log p ( x ( iK+m ) | θ − α∇θ 1 K ∑ n − log p ( x ( in ) |θ ) ︸ ︷︷ ︸ θ′Ti ) ] , ( 2 ) where θ′Ti is the task-specific parameters after a single batch update by gradient descent from θ . The meta-learner can achieve the parameter θ , which can quickly adapt to new tasks with a small amount of data by optimizing Equation 2 using the gradient method . We note that θ can be interpreted as the parameters of the prior distribution for the task-specific parameters θTi . By replacing the expectation w.r.t . the original posterior distribution by the maximum likelihood estimate∫ f ( θT ) p ( θT |θ ) dθT ' f ( θ′T ) , the abovementioned objective function ( Equation 2 ) recovers . 3 LOCAL VARIATIONAL AUTOENCODER . In this section , we will present the Local VAE , a variant of the VAE suitable for representation learning of a dataset with local structure . Here , we extend the objective function of the VAE ( Equation 1 ) to have different parameters for each local subset . We consider the variational lower bound of the log-likelihood for the dataset D , just as with the Vanilla VAE . However , we define the different model parameters θN ( x ( i ) ) and φN ( x ( i ) ) for each neighborhood N ( x ( i ) ) of the i-th data , respectively . Since these parameters are often highdimensional and require a long time and a large amount of data for training , we give meta parameters θ and φ as prior distributions of these local parameters . The overall model performs the probabilistic inference through the conditional distribution from the meta parameters . The variational lower bound for the log-likelihood can be calculated as follows : log pθ ( x ( i ) ) = log ∫ p ( x ( i ) |z , θN ( x ( i ) ) ) p ( z ) p ( θN ( x ( i ) ) |θ ) dzdθN ( x ( i ) ) ( 3 ) ≥ ∫ q ( z|x ( i ) , φN ( x ( i ) ) ) q ( φN ( x ( i ) ) |φ ) × log p ( x ( i ) |z , θN ( x ( i ) ) ) p ( z ) p ( θN ( x ( i ) ) |θ ) q ( z|x ( i ) , φN ( x ( i ) ) ) q ( φN ( x ( i ) ) |φ ) dzdθN ( x ( i ) ) dφN ( x ( i ) ) ( 4 ) =Eq ( z|x ( i ) , φ N ( x ( i ) ) ) q ( φ N ( x ( i ) ) |φ ) [ log p ( x ( i ) |z , θN ( x ( i ) ) ) p ( θN ( x ( i ) ) |θ ) ] −DKL ( q ( z|x ( i ) , φN ( x ( i ) ) ) q ( φN ( x ( i ) ) |φ ) ‖p ( z ) ) , ( 5 ) where p ( θN ( x ( i ) ) |θ ) and q ( φN ( x ( i ) ) |φ ) are the conditional distribution of the local parameters . We note that the integral variables of the expectation and the Kullback-Leibler divergence in Equation 5 are z , θN ( x ( i ) ) and φN ( x ( i ) ) . As we mentioned above , the integral variables of Equation 5 include θN ( x ( i ) ) and φN ( x ( i ) ) . This means that Equation 5 needs to take an integral of the model parameters to evaluate the objective function , while the one of Vanilla VAE only requires the Monte Carlo expectation of z . Such an integral is unreasonable in deep generative models where model parameters are often high-dimensional . To overcome this problem , we replace this integral with the maximum likelihood estimator updated by the one-step gradient method , as we described in Section 2.2 . Let L ( θ , φ ; x ( i ) ) be the negative of the expression obtained by Equation 5 . By replacing the integral of θN ( x ( i ) ) and φN ( x ( i ) ) with the maximum likelihood estimator θ′N ( x ( i ) ) and φ ′ N ( x ( i ) ) , we obtain L ( θ , φ ; x ( i ) ) '− Eq ( z|x ( i ) , φ′ N ( x ( i ) ) ) [ log p ( x ( i ) |z , θ′N ( x ( i ) ) ) ] +DKL ( q ( z|x ( i ) , φ′N ( x ( i ) ) ) ∥∥∥∥pθ ( z ) ) =Lg ( θ′N ( x ( i ) ) , φ ′ N ( x ( i ) ) ; x ( i ) ) . ( 6 ) We note that the integral variable of Equation 6 is now only z . The maximum likelihood estimator of the local parameters can be obtained by the following update rule : θ′N ( x ( i ) ) =θ − α∇θ 1 K ∑ x∈N ( x ( i ) ) L ( θ , φ ; x ) , ( 7 ) φ′N ( x ( i ) ) =φ− α∇φ 1 K ∑ x∈N ( x ( i ) ) L ( θ , φ ; x ) , ( 8 ) where K is the number of neighborhoods for x ( i ) . L ( θ , φ ; x ) in the above equations is the ELBO of Vanilla VAE defined in Equation 1 . Algorithm 1 shows the overall algorithm . From the perspective of the graphical model , our proposed Local VAE algorithm corresponds to the assumption that the dataset approximately lies on multiple subsets and that each subset is generated from different parameters . Alternatively , from the viewpoint of meta-learning , our objective function is consistent with the case of training VAE by MAML when task information is given as a neighbor graph . We can also give the relationship of our model to the conventional local learning approach . Please see Appendix A for the detail .

Summary: Incorporating the local learning approaches into the training of the deep generative models can potentially create a new model that has both the capacity for high-dimensional inputs and flexibility for locally changing environments. However, the local learning approach is limited to using only a simple model, because complex models require a large amount of data and extended training time. This paper overcomes this trade-off based on the insight that the real-world dataset often shares some structural similarities between each neighborhood.