Generative Restricted Kernel Machines

1 INTRODUCTION . In the past decade , interest in generative models has grown tremendously , finding applications in multiple fields such as , generated art , on-demand video , image denoising ( Vincent et al. , 2010 ) , exploration in reinforcement learning ( Florensa et al. , 2018 ) , collaborative filtering ( Salakhutdinov et al. , 2007 ) , inpainting ( Yeh et al. , 2017 ) and many more . Some examples of graphical models based on a probabilistic framework with latent variables are Variational Auto-Encoders ( Kingma & Welling , 2014 ) and Restricted Boltzmann Machines ( RBMs ) ( Smolensky , 1986 ; Salakhutdinov & Hinton , 2009 ) . More recently proposed models are based on adversarial training such as Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and its many variants . Furthermore , auto-regressive models such as Pixel Recurrent Neural Networks ( PixelRNNs ) ( Van Den Oord et al. , 2016 ) model the conditional distribution of every individual pixel given previous pixels . All these approaches have their own advantages and disadvantages . For example , RBMs perform both learning and Bayesian inference in graphical models with latent variables . However , such probabilistic models must be properly normalized , which requires evaluating intractable integrals over the space of all possible variable configurations ( Salakhutdinov & Hinton , 2009 ) . Currently GANs are considered as the state-of-the-art for generative modeling tasks , producing high-quality images but are more difficult to train due to unstable training dynamics , unless more sophisticated variants are applied . Many datasets are comprised of different representations of the data , or views . Views can correspond to different modalities such as sounds , images , videos , sequences of previous frames , etc . Although each view could individually be used for learning tasks , exploiting information from all views together could improve the learning quality ( Pu et al. , 2016 ; Liu & Tuzel , 2016 ; Chen & Denoyer , 2017 ) . Also , it is among the goals of the latent variable modelling to model the description of data in terms of uncorrelated or independent components . Some classical examples are Independent Component Analysis ; Hidden Markov models ( Rabiner & Juang , 1986 ) ; Probabilistic Principal Component Analysis ( PCA ) ( Tipping & Bishop , 1999 ) ; Gaussian-Process Latent variable model ( Lawrence , 2005 ) and factor analysis . Hence , when learning a latent space in generative models , it becomes interesting to find a disentangled representation . Disentangled variables are generally considered to contain interpretable information and reflect separate factors of variation in the data for e.g . lighting conditions , style , colors , etc . The definition of disentanglement in the literature is not precise , however many believe that a representation with statistically independent variables is a good starting point ( Schmidhuber , 1992 ; Ridgeway , 2016 ) . Such representations extract information into a compact form which makes it possible to generate samples with specific characteristics ( Chen et al. , 2018 ; Bouchacourt et al. , 2018 ; Tran et al. , 2017 ; Chen et al. , 2016 ) . Additionally , these representations have been found to generalize better and be more robust against adversarial attacks ( Alemi et al. , 2017 ) . In this work , we propose an alternative generative mechanism based on the framework of Restricted Kernel Machines ( RKMs ) ( Suykens , 2017 ) , called Generative RKM ( Gen-RKM ) . RKMs yield a representation of kernel methods with visible and hidden units establishing links between Kernel PCA , Least-Squares Support Vector Machines ( LS-SVM ) ( Suykens et al. , 2002 ) and RBMs . This framework has a similar energy form as RBMs , though there is a non-probabilistic training procedure where the eigenvalue decomposition plays the role of normalization . Recently , Houthuys & Suykens ( 2018 ) used this framework to develop tensor-based multi-view classification models and Schreurs & Suykens ( 2018 ) showed how kernel PCA fits into this framework . Contributions . 1 ) A novel multi-view generative model based on the RKM framework where multiple views of the data can be generated simultaneously . 2 ) Two methods are proposed for computing the pre-image of the feature vectors : with the feature map explicitly known or unknown . We show that the mechanism is flexible to incorporate both kernel-based , ( deep ) convolutional neural network based models within the same setting . 3 ) When using explicit feature maps , we propose a training algorithm that jointly performs the feature-selection and learns the common-subspace representation in the same procedure . 4 ) Qualitative and quantitative experiments demonstrate that the model is capable of generating good quality images of natural objects . Further experiments on multi-view datasets exhibit the potential of the model . Thanks to the orthogonality of eigenvectors of the kernel matrix , the learned latent variables are uncorrelated . This resembles a disentangled representation , which makes it possible to generate data with specific characteristics . This paper is organized as follows . In Section 2 , we discuss the Gen-RKM training and generation mechanism when multiple data sources are available . In Section 3 , we explain how the model incorporates both kernel methods and neural networks through the use of implicit and explicit feature maps respectively . When the feature maps are defined by neural networks , the Gen-RKM algorithm is explained in Section 4 . In Section 5 , we show experimental results of our model applied on various public datasets . Section 6 concludes the paper along with directions towards the future work . Additional supplementary materials are given in the Appendix A . 2 GENERATIVE RESTRICTED KERNEL MACHINES FRAMEWORK . The proposed Gen-RKM framework consists of two phases : a training phase and a generation phase which occurs one after another . 2.1 TRAINING . Similar to Energy-Based Models ( EBMs , see LeCun et al . ( 2004 ) for details ) , the RKM objective function captures dependencies between variables by associating a scalar energy to each configuration of the variables . Learning consists of finding an energy function in which the observed configurations of the variables are given lower energies than unobserved ones . Note that the schematic representation , as shown in Figure 1 is similar to Discriminative RBMs ( Larochelle & Bengio , 2008 ) and the objective function Jt ( defined below ) has an energy form similar to RBMs with additional regularization terms . The latent space dimension in the RKM setting has a similar interpretation as the number of hidden units in a restricted Boltzmann machine , where in the specific case of the RKM these hidden units are uncorrelated . We assume a datasetD = { xi , yi } Ni=1 , with xi ∈ Rd , yi ∈ Rp comprising ofN data points . Here yi may represent an additional view of xi , e.g. , an additional image from a different angle , the caption of an image or a class label . Starting from the RKM interpretation of Kernel PCA , which gives an upper bound on the equality constrained Least-Squares Kernel PCA objective function ( Suykens , 2017 ) , and applying the feature-maps φ1 : Rd 7→ Rdf and φ2 : Rp 7→ Rpf to the input data points , the training objective function Jt for generative RKM is given by1 : Jt = N∑ i=1 ( −φ1 ( xi ) > Uhi − φ2 ( yi ) > V hi + λ 2 h > i hi ) + η1 2 Tr ( U > U ) + η2 2 Tr ( V > V ) , ( 1 ) where U ∈ Rdf×s and V ∈ Rpf×s are the unknown interaction matrices , and hi ∈ Rs are the latent variables modeling a common subspaceH between the two input spaces X and Y ( see Figure 1 ) . The derivation of this objective function is given in the Appendix A.1 . Given η1 > 0 and η2 > 0 as regularization parameters , the stationary points of Jt are given by : ∂Jt ∂hi = 0 =⇒ λhi = U > φ1 ( xi ) + V > φ2 ( yi ) , ∀i = 1 , . . . , N ∂Jt ∂U = 0 =⇒ U = 1η1 ∑N i=1 φ1 ( xi ) h > i ∂Jt ∂V = 0 =⇒ V = 1η2 ∑N i=1 φ2 ( yi ) h > i . ( 2 ) Substituting U and V in the first equation above , denoting Λ = diag { λ1 , . . . , λs } ∈ Rs×s with s ≤ N , yields the following eigenvalue problem : [ 1 η1 K1 + 1 η2 K2 ] H > = H > Λ , ( 3 ) whereH = [ h1 , . . . , hN ] ∈ Rs×N with s ≤ N is the number of selected principal components and K1 , K2 ∈ RN×N are the kernel matrices corresponding to data sources2 . Based on Mercer ’ s theorem ( Mercer , 1909 ) , positive-definite kernel functions k1 : Rd × Rd 7→ R , k2 : Rp × Rp 7→ R can be defined such that k1 ( xi , xj ) = 〈φ1 ( xi ) , φ1 ( xj ) 〉 , and k2 ( yi , yj ) = 〈φ2 ( yi ) , φ2 ( yj ) 〉 , ∀i , j = 1 , . . . , N forms the elements of corresponding kernel matrices . The feature maps φ1 and φ2 , mapping the input data to the high-dimensional feature space ( possibly infinite ) are implicitly defined by kernel functions . Typical examples of such kernels are given by the Gaussian RBF kernel k ( xi , xj ) = e −‖xi−xj‖22/ ( 2σ2 ) or the Laplace kernel k ( xi , xj ) = e−‖xi−xj‖2/σ just to name a few ( Scholkopf & Smola , 2001 ) . However , one can also define explicit feature maps , still preserving the positive-definiteness of the kernel function by construction ( Suykens et al. , 2002 ) . 2.2 GENERATION . In this section , we derive the equations for the generative mechanism . RKMs resembling energybased models , the inference consists in clamping the value of observed variables and finding configurations of the remaining variables that minimizes the energy ( LeCun et al. , 2004 ) . Given the 1For convenience , it is assumed that all the feature vectors are centered in the feature spaceF using φ̃ ( x ) : = φ ( x ) − 1 N ∑N i=1 φ ( xi ) . Otherwise , a centered kernel matrix could be obtained using Eq . 17 ( Appendix A.4 ) . 2While in the above section we have assumed that only two data sources ( namely X and Y ) are available for learning , the above procedure could be extended to multiple data-sources . For theM views or data-sources , this yields the training problem : [ ∑M ` =1 1 η ` K ` ] H > = H > Λ . learned interconnection matrices U and V , and a given latent variable h ? , consider the following objective function : Jg = −φ1 ( x ? ) > Uh ? − φ2 ( y ? ) > V h∗ + 1 2 φ1 ( x ? ) > φ1 ( x ? ) + 1 2 φ2 ( y ? ) > φ2 ( y ? ) , ( 4 ) with an additional regularization term on data sources . Here Jg denotes the objective function for generation . The given latent variable h ? can be the corresponding latent code of a training point , a newly sampled hidden unit or a specifically determined one . Above cases correspond to generating the reconstructed visible unit , generating a random new visible unit or exploring the latent space by carefully selecting hidden units respectively . The stationary points of Jg are characterized by : { ∂Jg ∂φ1 ( x ? ) = 0 =⇒ φ1 ( x ? ) = Uh ? , ∂Jg ∂φ2 ( y ? ) = 0 =⇒ φ2 ( y ? ) = V h ? . ( 5 ) Using U and V from Eq . 2 , we obtain the generated feature vectors : φ1 ( x ? ) = ( 1 η1 N∑ i=1 φ1 ( xi ) h > i ) h ? , φ2 ( y ? ) = ( 1 η2 N∑ i=1 φ2 ( yi ) h > i ) h ? . ( 6 ) To obtain the generated data , one now needs to compute the inverse images of the feature maps φ1 ( · ) and φ2 ( · ) in the respective input spaces , i.e. , solve the pre-image problem . We seek to find the functions ψ1 : Rdf 7→ Rd and ψ2 : Rpf 7→ Rp corresponding to the two data-sources , such that ( ψ1 ◦φ1 ) ( x ? ) ≈ x ? and ( ψ2 ◦φ2 ) ( y ? ) ≈ y ? , where φ1 ( x ? ) and φ2 ( y ? ) are calculated using Eq . 6 . When using kernel methods , explicit feature maps are not necessarily known . Commonly used kernels such as the radial-basis function and polynomial kernels map the input data to a very high dimensional feature space . Hence finding the pre-image , in general , is known to be an ill-conditioned problem ( Mika et al. , 1999 ) . However , various approximation techniques have been proposed ( Bui et al. , 2019 ; Kwok & Tsang , 2003 ; Honeine & Richard , 2011 ; Weston et al. , 2004 ) which could be used to obtain the approximate pre-image x̂ of φ1 ( x ? ) . In section 3.1 , we employ one such technique to demonstrate the applicability in our model , and consequently generate the multi-view data . One could also define explicit pre-image maps . In section 3.2 , we define parametric pre-image maps and learn the parameters by minimizing the appropriately defined objective function . The next section describes the above two pre-image methods for both cases , i.e. , when the feature map is explicitly known or unknown , in greater detail .

Summary: This is a very good paper, building on the idea of Restricted Kernel Machines (drawing a nice parallel between Restricted Bolzman Machines and tools available in the Kernel modelling literature). In this manuscript, the author(s) extend the work to a generative model setting to achieve multi-view generation -- a generative model that can explain correlated variables from a common subspace. The manuscript is well-written and easy to follow and the algorithmic details are clear. Image generation is illustrated on standard datasets (MNIST / CIFAR / CelebA).