Title: GENERATIVE RESTRICTED KERNEL MACHINES

Abstract: We introduce a novel framework for generative models based on Restricted Kernel Machines (RKMs) with multi-view generation and uncorrelated feature learning capabilities, called Gen-RKM. To incorporate multi-view generation, this mechanism uses a shared representation of data from various views. The mechanism is flexible to incorporate both kernel-based, (deep) neural network and Convolutional based models within the same setting. To update the parameters of the network, we propose a novel training procedure which jointly learns the features and shared subspace representation. The latent variables are given by the eigendecomposition of the kernel matrix, where the mutual orthogonality of eigenvectors represents uncorrelated features. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of generated samples on various standard datasets.

1. INTRODUCTION:

In the past decade, interest in generative models has grown tremendously, finding applications in multiple fields such as, generated art, on-demand video, image denoising (Vincent et al., 2010) , exploration in reinforcement learning (Florensa et al., 2018) , collaborative filtering (Salakhutdinov et al., 2007) , inpainting (Yeh et al., 2017 ) and many more. Some examples of graphical models based on a probabilistic framework with latent variables are Variational Auto-Encoders (Kingma & Welling, 2014) and Restricted Boltzmann Machines (RBMs) (Smolensky, 1986; Salakhutdinov & Hinton, 2009) . More recently proposed models are based on adversarial training such as Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) and its many variants. Furthermore, auto-regressive models such as Pixel Recurrent Neural Networks (PixelRNNs) (Van Den Oord et al., 2016) model the conditional distribution of every individual pixel given previous pixels. All these approaches have their own advantages and disadvantages. For example, RBMs perform both learning and Bayesian inference in graphical models with latent variables. However, such probabilistic models must be properly normalized, which requires evaluating intractable integrals over the space of all possible variable configurations (Salakhutdinov & Hinton, 2009) . Currently GANs are considered as the state-of-the-art for generative modeling tasks, producing high-quality images but are more difficult to train due to unstable training dynamics, unless more sophisticated variants are applied.

Many datasets are comprised of different representations of the data, or views. Views can correspond to different modalities such as sounds, images, videos, sequences of previous frames, etc. Although each view could individually be used for learning tasks, exploiting information from all views together could improve the learning quality (Pu et al., 2016; Liu & Tuzel, 2016; Chen & Denoyer, 2017) . Also, it is among the goals of the latent variable modelling to model the description of data in terms of uncorrelated or independent components. Some classical examples are Independent Component Analysis; Hidden Markov models (Rabiner & Juang, 1986) ; Probabilistic Principal Component Analysis (PCA) (Tipping & Bishop, 1999) ; Gaussian-Process Latent variable model (Lawrence, 2005) and factor analysis. Hence, when learning a latent space in generative models, it becomes interesting to find a disentangled representation. Disentangled variables are generally considered to contain interpretable information and reflect separate factors of variation in the data for e.g. lighting conditions, style, colors, etc. The definition of disentanglement in the literature is not precise, however many believe that a representation with statistically independent variables is a good starting point (Schmidhuber, 1992; Ridgeway, 2016) . Such representations extract information into a compact form which makes it possible to generate samples with specific characteristics (Chen et al., 2018; Bouchacourt et al., 2018; Tran et al., 2017; Chen et al., 2016) . Additionally, these representations have been found to generalize better and be more robust against adversarial attacks (Alemi et al., 2017) .

In this work, we propose an alternative generative mechanism based on the framework of Restricted Kernel Machines (RKMs) (Suykens, 2017) , called Generative RKM (Gen-RKM). RKMs yield a representation of kernel methods with visible and hidden units establishing links between Kernel PCA, Least-Squares Support Vector Machines (LS-SVM) (Suykens et al., 2002) and RBMs. This framework has a similar energy form as RBMs, though there is a non-probabilistic training procedure where the eigenvalue decomposition plays the role of normalization. Recently, Houthuys & Suykens (2018) used this framework to develop tensor-based multi-view classification models and Schreurs & Suykens (2018) showed how kernel PCA fits into this framework.

Contributions. 1) A novel multi-view generative model based on the RKM framework where multiple views of the data can be generated simultaneously. 2) Two methods are proposed for computing the pre-image of the feature vectors: with the feature map explicitly known or unknown. We show that the mechanism is flexible to incorporate both kernel-based, (deep) convolutional neural network based models within the same setting. 3) When using explicit feature maps, we propose a training algorithm that jointly performs the feature-selection and learns the common-subspace representation in the same procedure. 4) Qualitative and quantitative experiments demonstrate that the model is capable of generating good quality images of natural objects. Further experiments on multi-view datasets exhibit the potential of the model. Thanks to the orthogonality of eigenvectors of the kernel matrix, the learned latent variables are uncorrelated. This resembles a disentangled representation, which makes it possible to generate data with specific characteristics. This paper is organized as follows. In Section 2, we discuss the Gen-RKM training and generation mechanism when multiple data sources are available. In Section 3, we explain how the model incorporates both kernel methods and neural networks through the use of implicit and explicit feature maps respectively. When the feature maps are defined by neural networks, the Gen-RKM algorithm is explained in Section 4. In Section 5, we show experimental results of our model applied on various public datasets. Section 6 concludes the paper along with directions towards the future work. Additional supplementary materials are given in the Appendix A.

2. GENERATIVE RESTRICTED KERNEL MACHINES FRAMEWORK:

The proposed Gen-RKM framework consists of two phases: a training phase and a generation phase which occurs one after another.

2.1. TRAINING:

Similar to Energy-Based Models (EBMs, see LeCun et al. (2004) for details), the RKM objective function captures dependencies between variables by associating a scalar energy to each configuration of the variables. Learning consists of finding an energy function in which the observed configurations of the variables are given lower energies than unobserved ones. Note that the schematic representation, as shown in Figure 1 is similar to Discriminative RBMs (Larochelle & Bengio, 2008) and the objective function J t (defined below) has an energy form similar to RBMs with additional regularization terms. The latent space dimension in the RKM setting has a similar interpretation as the number of hidden units in a restricted Boltzmann machine, where in the specific case of the RKM these hidden units are uncorrelated.

We assume a dataset D = {x i , y i } N i=1 , with x i ∈ R d , y i ∈ R p comprising of N data points. Here y i may represent an additional view of x i , e.g., an additional image from a different angle, the caption of an image or a class label. Starting from the RKM interpretation of Kernel PCA, which gives an upper bound on the equality constrained Least-Squares Kernel PCA objective function (Suykens, 2017) , and applying the feature-maps φ 1 : R d → R d f and φ 2 : R p → R p f to the input data points,

F x F y H X Y U U V V φ 1 (•) ψ 1 (•) ψ 2 (•) φ 2 (•)

Figure 1 : Gen-RKM schematic representation modeling a common subspace H between two data sources X and Y. The φ 1 , φ 2 are the feature maps (F x and F y represent the feature-spaces) corresponding to the two data sources. While ψ 1 , ψ 2 represent the pre-image maps. The interconnection matrices U , V model dependencies between latent variables and the mapped data sources.

the training objective function J t for generative RKM is given by 1 :

EQUATION

where U ∈ R d f ×s and V ∈ R p f ×s are the unknown interaction matrices, and h i ∈ R s are the latent variables modeling a common subspace H between the two input spaces X and Y (see Figure 1 ). The derivation of this objective function is given in the Appendix A.1.

Given η 1 > 0 and η 2 > 0 as regularization parameters, the stationary points of J t are given by:

EQUATION

Substituting U and V in the first equation above, denoting Λ = diag{λ 1 , . . . , λ s } ∈ R s×s with s ≤ N , yields the following eigenvalue problem:

EQUATION

where H = h 1 , . . . , h N ∈ R s×N with s ≤ N is the number of selected principal components and K 1 , K 2 ∈ R N ×N are the kernel matrices corresponding to data sources 2 . Based on Mercer's theorem (Mercer, 1909) , positive-definite kernel functions

k 1 : R d × R d → R, k 2 : R p × R p → R can be defined such that k 1 (x i , x j ) = φ 1 (x i ), φ 1 (x j )

, and k 2 (y i , y j ) = φ 2 (y i ), φ 2 (y j ) , ∀i, j = 1, . . . , N forms the elements of corresponding kernel matrices. The feature maps φ 1 and φ 2 , mapping the input data to the high-dimensional feature space (possibly infinite) are implicitly defined by kernel functions. Typical examples of such kernels are given by the Gaussian RBF kernel

k(x i , x j ) = e -xi-xj 2 2 /(2σ 2 ) or the Laplace kernel k(x i , x j ) = e -xi-xj 2/σ

just to name a few (Scholkopf & Smola, 2001 ). However, one can also define explicit feature maps, still preserving the positive-definiteness of the kernel function by construction (Suykens et al., 2002) .

2.2. GENERATION:

In this section, we derive the equations for the generative mechanism. RKMs resembling energybased models, the inference consists in clamping the value of observed variables and finding configurations of the remaining variables that minimizes the energy (LeCun et al., 2004) . Given the 1 For convenience, it is assumed that all the feature vectors are centered in the feature space F using φ(x) := φ(x) -1 N N i=1 φ(xi). Otherwise, a centered kernel matrix could be obtained using Eq. 17 (Appendix A.4). 2 While in the above section we have assumed that only two data sources (namely X and Y) are available for learning, the above procedure could be extended to multiple data-sources. For the M views or data-sources, this yields the training problem:

M =1 1 η K H = H Λ.

learned interconnection matrices U and V , and a given latent variable h , consider the following objective function:

EQUATION

with an additional regularization term on data sources. Here J g denotes the objective function for generation. The given latent variable h can be the corresponding latent code of a training point, a newly sampled hidden unit or a specifically determined one. Above cases correspond to generating the reconstructed visible unit, generating a random new visible unit or exploring the latent space by carefully selecting hidden units respectively. The stationary points of J g are characterized by:

EQUATION

Using U and V from Eq. 2, we obtain the generated feature vectors:

EQUATION

To obtain the generated data, one now needs to compute the inverse images of the feature maps φ 1 (•) and φ 2 (•) in the respective input spaces, i.e., solve the pre-image problem. We seek to find the functions ψ 1 : R d f → R d and ψ 2 : R p f → R p corresponding to the two data-sources, such that

(ψ 1 • φ 1 )(x ) ≈ x and (ψ 2 • φ 2 )(y ) ≈ y

, where φ 1 (x ) and φ 2 (y ) are calculated using Eq. 6.

When using kernel methods, explicit feature maps are not necessarily known. Commonly used kernels such as the radial-basis function and polynomial kernels map the input data to a very high dimensional feature space. Hence finding the pre-image, in general, is known to be an ill-conditioned problem (Mika et al., 1999) . However, various approximation techniques have been proposed (Bui et al., 2019; Kwok & Tsang, 2003; Honeine & Richard, 2011; Weston et al., 2004) which could be used to obtain the approximate pre-image x of φ 1 (x ). In section 3.1, we employ one such technique to demonstrate the applicability in our model, and consequently generate the multi-view data. One could also define explicit pre-image maps. In section 3.2, we define parametric pre-image maps and learn the parameters by minimizing the appropriately defined objective function. The next section describes the above two pre-image methods for both cases, i.e., when the feature map is explicitly known or unknown, in greater detail.

3. IMPLICIT & EXPLICIT FEATURE MAP:



3.1. IMPLICIT FEATURE MAP:

As noted in the previous section, since x may not exist, we find an approximation x. A possible technique is shown by Schreurs & Suykens (2018) . Left multiplying Eq. 6 by φ 1 (x i ) and φ 2 (y i ) , ∀i = 1, . . . , N , we obtain:

EQUATION

where,

k x = [k(x 1 , x ), . . . , k(x N , x )]

represents the similarities between φ 1 (x ) and training data points in the feature space, and K 1 ∈ R N ×N represents the centered kernel matrix of X . Similar conventions follow for Y respectively. Using the kernel-smoother method (Hastie et al., 2001) , the pre-images are given by:

EQUATION

where k1 (x i , x ) and k2 (y i , y ) are the scaled similarities (see Eq. 8) between 0 and 1 and n r the number of closest points based on the similarity defined by kernels k1 and k2 .

3.2. EXPLICIT FEATURE MAP:

While using an explicit feature map, Mercer's theorem is still applicable due to the positive semidefiniteness of the kernel function by construction, thereby allowing the derivation of Eq. 3. In the experiments, we use a set of (convolutional) neural networks as the feature maps φ θ (•). Another (transposed convolutional) neural network is used for the pre-image map ψ ζ (•) (Dumoulin & Visin, 2016) . The network parameters {θ, ζ} are learned by minimizing the reconstruction errors defined by

L 1 (x, ψ 1 ζ 1 (φ 1 θ 1 (x))) and L 2 (y, ψ 2 ζ 2 (φ 2 θ 2 (y))).

In our experiments, we use the mean-squared

errors L 1 (x, ψ 1 ζ 1 (φ 1 θ 1 (x))) = 1 N N i=1 x i -ψ 1 ζ 1 (φ 1 θ 1 (x i )) 2 2 and L 2 (y, ψ 2 ζ 2 (φ 2 θ 2 (y))) = 1 N N i=1 y i -ψ 2 ζ 2 (φ 2 θ 2 (y i ))

2 2 , however, in principle, one can use any other loss appropriate to the dataset. Here φ 1 θ 1 (x i ) and φ 2 θ 2 (y i ) are computed from Eq. 6, i.e., the generated points in feature space from the subspace H.

Adding the loss function directly into the objective function J t is not suitable for minimization. Instead, we use the stabilized objective function defined as J stab = J t + c stab 2 J 2 t , where c stab ∈ R + is the regularization constant (Suykens, 2017) . This tends to push the objective function J t towards zero, which is also the case when substituting the solutions λ i , h i back into J t (see Appendix A.3 for details). The combined training objective is given by: min θ1,θ2,ζ1,ζ2

EQUATION

where c acc ∈ R + is a regularization constant to control the stability with reconstruction accuracy. In this way, we combine feature-selection and subspace learning within the same training procedure.

There is also an intuitive connection between Gen-RKM and autoencoders. Namely, the properties of kernel PCA resemble the objectives of the 3 variations of an autoencoder: standard (Kramer, 1991), VAE (Kingma & Welling, 2014) and β-VAE (Higgins et al., 2017) . 1) Similar to an autoencoder, Gen-RKM minimizes the reconstruction error in the loss function (see Eq. 9), where kernel PCA which acts as a denoiser (the information is compressed in the principal components). 2) By interpreting kernel PCA within the LS-SVM setting (Suykens et al., 2002) , the PCA analysis can take the interpretation of a one-class modeling problem with zero target value around which one maximizes the variance (Suykens et al., 2003) . When choosing a good feature map, one expects the latent variables to be normally distributed around zero. This property resembles the added regularization term in the objective of the VAE (Kingma & Welling, 2014), which is expressed as the Kullback-Leibler divergence between the encoder's distribution and a unit Gaussian as a prior on the latent variables. 3) Kernel PCA gives uncorrelated components in feature space. While it was already shown that PCA does not give a good disentangled representation for images (Eastwood & Williams, 2018; Higgins et al., 2017) . Hence by designing a good kernel (through appropriate feature-maps) and doing kernel PCA, it is possible to get a disentangled representation for images as we show on the example in Figure 5 . The uncorrelated components enhances the interpretation of the model.

4. THE GEN-RKM ALGORITHM:

Based on the previous analysis, we propose a novel algorithm, called the Gen-RKM algorithm, combining kernel learning and generative models. We show that this procedure is efficient to train and evaluate. It is also scalable to large datasets when using explicit feature maps. The training procedure simultaneously involves feature selection, common-subspace learning and pre-image map learning. This is achieved via an optimization procedure where one iteration involves an eigendecomposition of the kernel matrix which is composed of the features from various views (see Eq.

3). The latent variables are given by the eigenvectors, which are then passed via a pre-image map to reconstruct the sample. Figure 1 shows a schematic representation of the algorithm when two data sources are available.

Thanks to training in m mini-batches, this procedure is scalable to large datasets (sample size N ) with training time scaling super-linearly with T m = c N γ m γ-1 , instead of T k = cN γ , where γ ≈ 3 for algorithms based on decomposition methods, with some proportionality constant c. The training time could be further reduced by computing the covariance matrix (size

(d f +p f )×(d f +p f )) instead of a kernel matrix (size N m × N m )

, when the sum of the dimensions of the feature-spaces is less than the samples in mini-batch i.e. d f + p f ≤ N m . While using neural networks as feature maps, d f and p f correspond to the number of neurons in the output layer, which are chosen as hyperparameters by the practitioner. Eigendecomposition of this smaller covariance matrix would yield U and V as eigenvectors (see Eq. 10 and Appendix A.2 for detailed derivation), where computing the h i involves only matrix-multiplication which is readily parallelizable on modern GPUs: 

EQUATION

5. EXPERIMENTS:

To demonstrate the applicability of the proposed framework and algorithm, we trained the Gen-RKM model on a variety of datasets commonly used to evaluate generative models: MNIST (Le-Cun & Cortes, 2010), Fashion-MNIST (Xiao et al., 2017) , CIFAR-10 (Krizhevsky, 2009) , CelebA (Liu et al., 2015) , Dsprites (Matthey et al., 2017) and Teapot (Eastwood & Williams, 2018) . The experiments were performed using both the implicit feature map defined by a Gaussian kernel and parametric explicit feature maps defined by deep neural networks, either convolutional or fully connected. As explained in Section 2, in case of kernel methods, training only involves constructing the kernel matrix and solving the eigenvalue problem in Eq. 3. In our experiments, we fit a Gaussian mixture model (GMM) with l components to the latent variables of the training set, and randomly sample a new point h for generating views using a kernel smoother. In case of explicit feature maps, we define φ 1 θ 1 and ψ 1 ζ 1 as convolution and transposed-convolution neural networks, respectively (Dumoulin & Visin, 2016) ; and φ 2 θ 2 and ψ 1 ζ 2 as fully-connected networks. The particular architecture details are outlined in Table 3 in the Appendix. The training procedure in case of explicitly defined maps consists of minimizing J c using the Adam optimizer (Kingma & Ba, 2014) to update the weights and biases. To speed-up learning, we subdivided the datasets into m mini-batches, and within each iteration of the optimizer, Eq. 3 is solved to update the value of H. Information on the datasets and hyperparameters used for the experiments is given in Table 4 in the Appendix.

Generation::

Qualitative examples: Figure 2 shows the generated images using a convolutional neural network and transposed-convolutional neural network as the feature map and pre-image map respectively. The first column in yellow-boxes shows the training samples and the second column on the right shows the reconstructed samples. The other images shown are generated by random sampling from For a fair comparison, the models have the same encoder/decoder architecture, optimization parameters and are trained until convergence, where the details are given in Table 3 . We evaluate the performance qualitatively by comparing reconstruction and random sampling, the results are shown in Figure 8 in the Appendix. In order to quantitatively assess the quality of the randomly generated samples, we use the Fréchet Inception Distance (FID) introduced by Heusel et al. (2017) . The results are reported in Table 1 . Experiments were repeated for different latent-space dimensions (h dim ), and we observe empirically that FID scores are better for the Gen-RKM. This is confirmed by the qualitative evaluation in Table 8 , where the VAE generates smoother images. An interesting trend could be noted that as the dimension of latent-space is increased, VAE gets better at generating images whereas the performance of Gen-RKM decreases slightly. This is attributed to the eigendecomposition of the kernel matrix whose eigenvalue spectrum decreases rapidly depicting that most information is captured in few principal components, while the rest is noise. The presence of noise hinders the convergence of the model. It is therefore important to select the number of latent variables proportionally to the size of the mini-batch and the corresponding spectrum of the kernel matrix (the diversity within a mini-batch affects the eigenvalue spectrum of the kernel matrix). In these datasets, labels or attributes are seen as another view of the image that provides extra information. One-hot encoding of the labels was used to train the model. Figure 4a shows the generated images and labels when feature maps are only implicitly known i.e. through a Gaussian kernel. Figures 4b, 4c shows the same when using fully-connected networks as parametric functions to encode and decode labels. We can see that both the generated image and the generated label matches in most cases, albeit not all. 5 , where we explore the uncorrelated features learned by the models on the Dsprites and celebA dataset. In our experiments, the Dsprites training dataset comprised of 32 × 32 positions of oval and heart-shaped objects. The number of principal components chosen were 2 and the goal was to findout whether traversing along the eigenvectors, corresponds to traversing the generated im-age in one particular direction while preserving the shape of the object. Rows 1 and 2 of Figure 5 show the reconstructed images of an oval while moving along first and second principal component respectively. Notice that the first and second components correspond to the y and x positions respectively. Rows 3 and 4 show the same for hearts. On the celebA dataset, we train the Gen-RKM with 15 components. Rows 5 and 6 shows the reconstructed images while traversing along the principal components. When moving along the first component from left-to-right, the hair-color of the women changes, while preserving the face structure. Whereas traversal along the second component, transforms a man to woman while preserving the orientation. When the number of principal components were 2 while training, the brightness and background light-source corresponds to the two largest variances in the dataset. Also notice that, the reconstructed images are more blurry due to the selection of less number of components to model H.

Comparison: To quantitatively assess disentanglement performance, we compare Gen-RKM with VAE (Kingma & Welling, 2014) and beta-VAE (Higgins et al., 2017) on the Dsprites and Teapot datasets (Eastwood & Williams, 2018) . The models have the same encoder/decoder architecture, optimization parameters and are trained until convergence, where the details are given in Table 3 . The performance is measured using the proposed framework3 of Eastwood & Williams (2018) , which gives 3 measures: disentanglement, completeness and informativeness. The results are depicted in Table 2 . Gen-RKM has good performance on the Dsprites dataset when the latent space dimension is equal to 2. This is expected as the number of disentangled generating factors in the dataset is also equal to 2, hence there are no noisy components in the kernel PCA hindering the convergence. The opposite happens in the case h dim = 10, where noisy component are present. The above is confirmed by the Relative Importance Matrix on Figure 6 in the Appendix, where the 2 generating factors are well separated in the latent space of the Gen-RKM. For the Teapot dataset, Gen-RKM has good performance when h dim = 10. More components are needed to capture all variations in the dataset, where the number of generating factors is now equal to 5. In the other cases, Gen-RKM has a performance comparable to the others.

6. CONCLUSION AND FUTURE WORK:

The paper proposes a novel framework, called Gen-RKM, for generative models based on RKMs with extensions to multi-view generation and learning uncorrelated representations. This allows for a mechanism where the feature map can be implicitly defined using kernel functions or explicitly by (deep) neural network based methods. When using kernel functions, the training consists of only solving an eigenvalue problem. In the case of a (convolutional) neural network based explicit feature map, we used (transposed) networks as the pre-image functions. Consequently, a training procedure was proposed which involves joint feature-selection and subspace learning. Thanks to training in mini-batches and capability of working with covariance matrices, the training is scalable to large datasets. Experiments on benchmark datasets illustrate the merit of the proposed framework for generation quality as well as disentanglement. Extensions of this work consists of adapting the model to more advanced multi-view datatsets involving speech, images and texts; further analysis on other feature maps, pre-image methods, loss-functions and uncorrelated feature learning. Finally, this paper has demonstrated the applicability of the Gen-RKM framework, suggesting new research directions to be worth exploring.

A APPENDIX:

A.1 DERIVATION OF GEN-RKM OBJECTIVE FUNCTION Given D = {x i , y i } N i=1 , where x i ∈ R d , y i ∈ R p and feature-map φ 1 : R d → R d f and φ 2 : R p → R p f , the Least-Squares Support Vector Machine (LS-SVM) formulation of Kernel PCA (Suykens et al., 2002) for the two data sources can be written as:

EQUATION

where U ∈ R d×s and V ∈ R p×s are the interconnection matrices.

Using the notion of conjugate feature duality introduced in Suykens ( 2017), the error variables e i are conjugated to latent variables h i using:

EQUATION

which is also known as the Fenchel-Young inequality for the case of quadratic functions (Rockafellar, 1974) . By eliminating the variables e i from Eq. 11 and using Eq. 12, we obtain the Gen-RKM training objective function:

EQUATION

A.2 KERNEL PCA IN THE PRIMAL From Eq. 2, eliminating the variables h i yields the following:

EQUATION

Denote Φ x := [φ 1 (x 1 ), . . . , φ 1 (x N )], Φ y := [φ 2 (y 1 ), . . . , φ 2 (y N )] and Λ = diag{λ 1 , . . . , λ s } ∈ R s×s with s ≤ N . Now, composing the above equations in matrix form, we get the following eigen-decomposition problem:

EQUATION

)

Here the size of the covariance matrix is (d f + p f ) × (d f + p f ). The latent variables h i can be computed using Eq. 2, which simply involves matrix multiplications.

A.3 STABILIZING THE OBJECTIVE FUNCTION:

Proposition 1. All stationary solutions for H,Λ in Eq. 3 of J t lead to J t = 0.

Proof. Let λ i , h i are given by Eq. 3. Using Eq. 2 to substitute V and U in Eq. 1 yields:

J t (V , U , Λ, H) = N i=1 - λ 2 h i h i + η 1 2 Tr   1 η 2 1 N i=1 h i φ 1 (x i ) N j=1 φ 1 (x j )h j   + η 2 2 Tr   1 η 2 2 N i=1 h i φ 2 (y i ) N j=1 φ 2 (y j )h j   = N i=1 - λ 2 h i h i + η 1 2 Tr 1 η 2 1 HK 1 H + η 2 2 Tr 1 η 2 2 HK 2 H = N i=1 - λ 2 h i h i + 1 2 Tr H 1 η 1 K 1 + 1 η 2 K 2 H .

From Eq. 3, we get:

J t (V , U , Λ, H) = N i=1 - λ 2 h i h i + 1 2 Tr HH λ = N i=1 - λ 2 h i h i + λ 2 N i=1 h i h i = 0.

Proposition 2. Let J(x) : R N -→ R be a smooth function, for all x ∈ R N and for c ∈ R >0 , define J(x) := J(x) + c 2 J(x) 2 . Assuming (1 + cJ(x)) = 0, then x is the stationary points of J(x) iff

x is the stationary point for J(x).

Proof. Let x be a stationary point of J(x), meaning that ∇J(x ) = 0. The stationary points for J(x) can be obtained from:

EQUATION

It is easy to see from Eq. 2 that if x = x * , ∇J(x * ) = 0, we have that d J dx x * = 0, meaning that all the stationary points of J(x) are stationary points of J(x).

To show the other way, let x be stationary point of J(x) i.e. ∇ J(x ) = 0. Assuming (1 + cJ(x )) = 0, then from Eq. 16 for all c ∈ R >0 , we have (1 + cJ(x )) ∇J(x ) = 0, implying that ∇J(x ) = 0.

Based on the above propositions, we stabilize our original objective function Eq. 1 to keep it bounded and hence is suitable for minimization with Gradient-descent methods. Without the reconstruction errors, the stabilized objective function is

min U ,V ,h i J t + c 2 J 2 t .

Denoting J = J t + c stab 2 J 2 t . Since the derivatives of J t are given by Eq. 2, the stationary points of J are:

           ∂ J ∂V = (1 + c stab J t ) - N i=1 φ 1 (x i )h i + η 1 V = 0 =⇒ V = 1 η1 N i=1 φ 1 (x i )h i , ∂ J ∂U = (1 + c stab J t ) - N i=1 φ 2 (y i )h i + η 2 U = 0 =⇒ U = 1 η2 N i=1 φ 2 (y i )h i , ∂ J ∂hi = (1 + c stab J t ) -V φ 1 (x i ) -U φ 2 (y i ) + λh i = 0 =⇒ λh i = V φ 1 (x i ) + U φ 2 (y i ),

assuming 1 + c stab J t = 0. Elimination of V and U yields 1 η1 K 1 + 1 η2 K 2 H = H Λ, which is indeed the same solution for c stab = 0 in Eq. 1 and Eq. 3.

A.4 CENTERING OF KERNEL MATRIX:

Centering of the kernel matrix is done by the following equation:

EQUATION

where 1 denotes an N -dimensional vector of ones and K is either K 1 or K 2 .

:

Code and dataset available at https://github.com/cianeastwood/qedr

Figures and Tables:

Figure 2: Generated samples from the model using CNN as explicit feature map in the kernel function. In (a), (b), (c), (d) the yellow boxes in the first column show training examples and the adjacent boxes show the reconstructed samples. The other images (columns 3-6) are generated by random sampling from the fitted distribution over the learned latent variables. (e) and (f) shows the generated images through bilinear interpolations in the latent space.

MNIST: Implicit feature maps with Gaussian kernel are used during training. For generation, the pre-images are computed using the kernel-smoother method. MNIST: Explicit feature maps and the corresponding pre-image maps are defined by the Convolutional Neural Networks. horse bird horse dog cat plane horse bird car plane ship deer (c) CIFAR-10: Explicit feature maps as Convolutional Neural Networks. Pre-images are computed using Transposed CNNs.

Figure 4: Multi-view Generation (images and labels) on various datasets using implicit and explicit feature maps.

Figure 5: Exploring the learned uncorrelated-features by traversing along the eigenvectors. The first column shows the scatter plot of latent variables using the top two principal components. The green lines within, show the traversal in the latent space and the related rows show the corresponding reconstructed images.

)

Table content: <table><tr><td colspan="2">Algorithm 1 Gen-RKM</td><td/><td/></tr><tr><td colspan="4">Input: {xi, yi} N i=1 , η1, η2, feature map φj(•) -explicit or implicit via kernels kj(•, •), for j ∈ {1, 2}</td></tr><tr><td colspan="2">Output: Generated data x , y</td><td/><td/></tr><tr><td colspan="2">1: procedure TRAIN</td><td colspan="2">1: procedure GENERATION</td></tr><tr><td>2:</td><td>if φj(•) = Implicit then</td><td>2:</td><td>Select h</td></tr><tr><td>3:</td><td>Hyperparameters: kernel specific</td><td>3:</td><td>if φj(•) = Implicit then</td></tr><tr><td>4:</td><td>Solve Eq. 3</td><td>4:</td><td>Hyperparameter: nr</td></tr><tr><td>5:</td><td>Select s principal components</td><td>5:</td><td>Compute kx *  , ky *  (Eq. 7)</td></tr><tr><td>6:</td><td>else if φj(•) = Explicit then</td><td>6:</td><td>Get x, ŷ (Eq. 8)</td></tr><tr><td>7:</td><td>while not converged do</td><td>7:</td><td>else if φj(•) = Explicit then</td></tr><tr><td>8:</td><td>{x, y} ← {Get mini-batch}</td><td>8:</td><td>do steps 11-12</td></tr><tr><td>9:</td><td>φ1(x) ← x; φ2(y) ← y</td><td>9:</td><td>end if</td></tr><tr><td>10:</td><td>do steps 4-5</td><td colspan="2">10: end procedure</td></tr><tr><td>11:</td><td>{φ1(x), φ2(y)} ← h (Eq. 6)</td><td/><td/></tr><tr><td>12:</td><td>{x, y} ← {ψ1(φ1(x)), ψ2(φ2(y))}</td><td/><td/></tr><tr><td>13:</td><td>∆θ1 ∝ -∇ θ 1 Jc; ∆θ2 ∝ -∇ θ 2 Jc</td><td/><td/></tr><tr><td>14:</td><td>∆ζ1 ∝ -∇ ζ 1 Jc; ∆ζ2 ∝ -∇ ζ 2 Jc</td><td/><td/></tr><tr><td>15:</td><td>end while</td><td/><td/></tr><tr><td>16:</td><td>end if</td><td/><td/></tr><tr><td colspan="2">17: end procedure</td><td/><td/></tr></table>

FID Scores(Heusel et al., 2017) for randomly generated samples (smaller is better).

Table content: <table><tr><td colspan="2">Dataset Algorithm</td><td/><td>FID score</td><td/></tr><tr><td/><td/><td colspan="3">h dim = 10 h dim = 30 h dim = 50</td></tr><tr><td>MNIST</td><td>Gen-RKM VAE</td><td>89.825 250</td><td>130.497 234.749</td><td>131.696 205.282</td></tr><tr><td>CelebA</td><td>Gen-RKM VAE</td><td>103.299 286.039</td><td>84.403 245.738</td><td>85.121 225.783</td></tr></table>

Qualitative examples: The latent variables are uncorrelated, which gives an indication that the model could resemble a disentangled representation. This is confirmed by the empirical evidence on Figure

Table content: <table><tr><td>Lasso</td><td>Random Forest</td></tr></table>

Peer review (iclr2020):
There exists two papers: [1] Multimodal Learning with Deep Boltzmann Machines, http://jmlr.org/papers/volume15/srivastava14b/srivastava14b.pdf [2] Deep Restricted Kernel Machines Using Conjugate Feature Duality, ftp://ftp.esat.kuleuven.ac.be/stadius/suykens/reports/deepRKM1.pdf In particular [2] considers a model, which is similar to a Boltzmann machine, but at the same time it is based on kernel features, and uses structure of the corresponding optimization problem to obtain a solution in a semi-explicit way. The authors of the considered paper 1) generalise a multimodal variant of the Boltzmann machine from [1] (which uses a special cross-product term to take into account dependency between modalities) to the case of kernel machines, 2) demonstrate on several typical datasets that using explicit deep network features it is possible to model images and data with two modalities (faces/textual description of faces). Comments: - From the description of the functionals L1 and L2 (bottom of page 4 and top of the page 5) the reader can think that the authors tune parameters (zeta_1,theta_1) and (zeta_2,theta_2) for each sample point separately - The authors claimed that the experiments were done both for kernel features and for explicit features based on neural networks. However, in the experimental section there are no results obtained when using implicit kernels. Nothings is told on how to select kernel parameters - The authors claimed that thanks to PCA-like definition of latent vectors they are orthogonal which is similar to disentangle representations. However, there are no any empirical evidences whether it is possible to benefit somehow from that orthogonal property, as well as there is no comparison with approaches to construct disentangle? latent representation for other types of generative models. Conclusions: - In general the text is accurately written, the work is well organised. - Still I was not able to understand the main idea of the paper. a) if the main idea of the paper that the authors propose some new method for generative modeling of multi-modal data, then the authors should make significantly more diverse experiments and ablation studies. Actually, this is not the case of the current work. The authors did not provide any quantitate measure and comparison with existing approaches; b) if the main idea is to present a new approach, then still I would not call the approach completely new, as it is based on well-known ideas and its benefits are completely not obvious. I guess that the paper can be published, but only after issues in a) are addressed.